This is a breakdown of what I learned while getting it to work, from the initial setup to the weird debugging hurdles you'll likely hit.

Core Resources

These are the docs I kept open the whole time. They’re the best source of truth right now:

• OpenAI Apps SDK Quickstart: Start here.

• Building an MCP Server: The technical details of the protocol.

• App Submission Guidelines: Good to read early so you don't have to rewrite things later.

How it works: The Architecture

There are two main parts to a ChatGPT App:

1. Tools: These are the actions ChatGPT can take. When a user asks for something, ChatGPT sends a JSON request to your /mcp endpoint.

2. Resources: This is the UI. Your tool returns a URI, and ChatGPT fetches the HTML to render it in a sandbox.

The Flow:

• User prompt triggers a tool call.

• Your server runs the handler and returns structuredContent (data for the widget) and _meta.

• ChatGPT loads the HTML (served as text/html+skybridge).

• The Widget Runtime passes the data into your code via window.openai.

The Iframe Approach: The ChatGPT sandbox is very restrictive. To get around this, I serve a tiny HTML wrapper that just loads my "real" app inside an <iframe>. This lets me use my normal tech stack while still living inside the ChatGPT interface.

Setup & Tunneling

Dependencies

The MCP SDK is moving fast, so I'd recommend pinning these versions to avoid breaking changes:

npm install mcp-handler@1.0.2 @modelcontextprotocol/sdk@1.20.0

Note: There is a Vercel starter out there, but I found it easier to build from scratch since the SDK updates so frequently. (https://github.com/vercel-labs/chatgpt-apps-sdk-nextjs-starter)

Local Development

ChatGPT needs an HTTPS endpoint to talk to. I use a VPS with nginx and an SSH tunnel to point traffic to my local machine. So basically it does : ChatGPT → Cloudflare redirects DNS like XXXX.domain.com → VPS → Nginx to specific local port → My local machine with a SSH reverse proxy.

The Tunnel:

# Run this locally to map your VPS port to your local dev server
ssh -R 8888:localhost:3000 -N -o ServerAliveInterval=30 user@your-vps-ip

Next.js Config

You need to set specific headers so OpenAI can embed your app. Here’s what my next.config.js looks like:

async headers() {
  return [
    {
      source: '/mcp',
      headers: [
        { key: 'Access-Control-Allow-Origin', value: '*' },
        { key: 'Access-Control-Allow-Methods', value: 'GET,POST,OPTIONS' },
        { key: 'Access-Control-Allow-Headers', value: '*' },
      ],
    },
    {
      source: '/:path*', 
      headers: [
        { key: 'X-Frame-Options', value: 'ALLOWALL' },
        {
          key: 'Content-Security-Policy',
          value: "frame-ancestors 'self' https://*.openai.com https://*.oaiusercontent.com https://*.web-sandbox.oaiusercontent.com"
        },
      ],
    },
  ];
}

The MCP Route Handler

In Next.js, you'll handle everything in app/mcp/route.ts. This is where you define your tools and resources.

import { createMcpHandler } from "mcp-handler";
import { z } from "zod";

const handler = createMcpHandler(async (server) => {
  // 1. The UI Resource
  server.registerResource(
    "my-app-ui",
    "ui://widget/view.html",
    {
      title: "App View",
      mimeType: "text/html+skybridge", 
      _meta: {
        "openai/widgetPrefersBorder": true,
        "openai/widgetDescription": "Interactive viewer",
        "openai/widgetDomain": "https://your-app.com",
        "openai/widgetCSP": {
          connect_domains: ["https://api.your-app.com"],
          resource_domains: ["https://static.your-app.com"],
          frame_domains: ["https://your-app.com"], 
        }
      }
    },
    async () => ({
      contents: [{
        uri: "ui://widget/view.html",
        mimeType: "text/html+skybridge",
        text: createIframeWrapper("https://your-app.com/embed"),
      }]
    })
  );

  // 2. The Tool
  server.registerTool(
    "create_something",
    {
      description: "Creates a visualization",
      inputSchema: { prompt: z.string() },
      _meta: {
        "openai/outputTemplate": "ui://widget/view.html",
        "openai/toolInvocation/invoking": "Generating...",
        "openai/toolInvocation/invoked": "Ready!",
        "openai/resultCanProduceWidget": true,
        "openai/widgetAccessible": true,
      }
    },
    async ({ prompt }) => ({
      content: [{ type: "text", text: `Generating: ${prompt}` }],
      structuredContent: { prompt }, 
      _meta: { "openai/outputTemplate": "ui://widget/view.html" }
    })
  );
});

export const GET = handler;
export const POST = handler;

The Iframe Wrapper & Widget Runtime

The createIframeWrapper function returns the HTML that ChatGPT renders. It grabs the data from ChatGPT and passes it to your app via URL parameters.

Using window.openai

Inside the sandbox, you get a window.openai object. This is how you talk to the host:

• toolOutput: The data your tool just returned.

• widgetState: State that persists between turns.

• callTool(name, args): Trigger other tools from your UI.

• openExternal(url): Open links outside the chat.

function createIframeWrapper(appUrl) {
  return `
    <!DOCTYPE html>
    <html>
      <body style="margin:0; padding:0; overflow:hidden;">
        <iframe id="frame" style="width:100vw; height:100vh; border:none;"></iframe>
        <script>
          const frame = document.getElementById('frame');

          // Get data from OpenAI
          const output = window.openai?.toolOutput || {};
          const prompt = output.structuredContent?.prompt || '';
          frame.src = \`\${appUrl}?prompt=\${encodeURIComponent(prompt)}\`;

          // Listen for updates if the user asks for changes
          window.addEventListener('openai:set_globals', (e) => {
            const newPrompt = e.detail?.globals?.toolOutput?.structuredContent?.prompt;
            if (newPrompt) frame.src = \`\${appUrl}?prompt=\${encodeURIComponent(newPrompt)}\`;
          });
        </script>
      </body>
    </html>
  `;
}

Development Workflow

Testing can be slow if you keep starting new chats. Here’s how I sped it up:

1. Developer Mode: Enable this in ChatGPT settings to add your local server.

2. The Reload Button: You don't need a new chat for every change. Just click the "Reload" icon on your app in the sidebar to refresh tool definitions.

3. Inspect Element: You can right-click the widget in ChatGPT and "Inspect" it. This is the only way to see console logs from your wrapper.

Common Pitfalls

• Blank Widgets: Usually a CSP issue. Check the console for frame-ancestors errors.

• CORS Errors: ChatGPT sends OPTIONS requests. If your proxy or server isn't handling preflight, it will fail.

• MIME Types: If you don't use text/html+skybridge, ChatGPT will just show your HTML as a code block.

• Auth: If your app needs a login, the iframe might get stuck. I usually serve a public "viewer" route and pass a token in the URL.

Submission & Launch

When you're ready to submit to the OpenAI App Store, keep these in mind:

Metadata Hints

OpenAI uses these to decide how to treat your tools:

• readOnlyHint: For tools that don't change data.

• openWorldHint: For tools that touch external APIs.

• destructiveHint: For tools that delete things (triggers a confirmation).

Requirements

1. Domain Verification: You need to host a token at /.well-known/openai-apps-challenge.

2. Tool Descriptions: Be very specific. The model needs to know exactly when to use your tool.

3. Privacy Policy: You'll need a hosted policy on your domain.

Wrapping Up

Building an MCP app is a bit different from standard web dev, but once you get the tunneling and the iframe wrapper working, it opens up a lot of possibilities.

If you're looking for more examples, check out the Vercel MCP Starter or the official OpenAI docs.

Happy building!

Theo - Founder of Datadef

The sensor had actually detected the anomaly. The temperature readings showed a clear upward trend starting six hours before the failure. But somewhere between the raw sensor data and the predictive model, we'd lost the signal. A validation rule, one I'd written myself trying to be clever about filtering outliers, had flagged the rising temperatures as "implausible" and excluded them from the clean dataset. The model never saw the warning signs because I'd accidentally taught the system to ignore them.

That night taught me something no architecture diagram ever could: in manufacturing, the cost of being too aggressive with data cleaning is measured in broken machinery and production downtime. And it completely reshaped how I approach industrial IoT pipelines.

SCADA Data is different from everything else

Let me tell you about SCADA data, because until you've worked with it, you don't really understand why factory floor data is different from everything else in the data engineering world.

SCADA systems were designed in an era when storage was expensive and networks were unreliable. They compress aggressively, they batch opportunistically, and they have their own ideas about what timestamps mean. I once spent two weeks debugging a data quality issue that turned out to be a SCADA system reporting timestamps in local time during winter and UTC during summer — not because of daylight saving time, but because someone had configured it that way a decade ago and nobody remembered why.

The instinct when you first encounter this is to normalize everything immediately. Parse the timestamps, convert to UTC, standardize the schemas. Get it into a clean format as fast as possible so you can work with it like normal data. This instinct is wrong, and it took me years to understand why.

The problem is that normalization destroys information. When you convert that quirky timestamp to UTC, you lose the ability to debug timestamp-related issues. When you parse a numeric value from a string, you can no longer see that the source was sending "23.5°C" with a unit suffix that your parser silently dropped. When you unify SCADA data and PLC data into a single table because "they're both sensor readings," you lose the ability to reason about their different reliability characteristics.

I now keep SCADA and PLC data completely separate all the way through ingestion. They land in different tables, with different schemas, preserving all the quirks of their respective sources. When something goes wrong, I can look at exactly what each system sent, in exactly the format it was sent, without wondering whether my transformation logic introduced the bug.

This might sound obvious, but I've reviewed dozens of industrial data architectures, and at least half of them try to unify everything too early. "But it's more efficient!" they say. "We don't want to maintain multiple pipelines!" And then they spend weeks debugging issues that would have been obvious if they could just see what the source systems actually sent.

Flag, Don't Filter

The real art in industrial data isn't ingestion. It's knowing when to validate and when to preserve.

Here's the tension: sensors lie. They drift out of calibration, they malfunction, they send garbage when there's electrical interference. You absolutely need validation logic to catch these issues. But sensors also tell the truth in ways that look like lies. A temperature spike that seems "implausible" might be the first sign of a bearing failure. A pressure reading outside normal ranges might indicate a process change that operations made without telling anyone.

The validation rule that bit me at 5 AM was checking whether temperature readings fell within two standard deviations of the rolling average. Statistically sound, right? The problem is that equipment failure doesn't follow normal distributions. It starts with small anomalies that grow over time, exactly the kind of gradual deviation that a rolling-average filter is designed to smooth out.

I've since moved to a different approach: flag, don't filter. When a reading looks suspicious, I add a warning flag and a severity level, but I keep the data. The downstream models can decide whether to include or exclude flagged readings based on their specific use case. Real-time alerting might want to include everything and tolerate some false positives. Historical analysis for predictive maintenance definitely wants the "anomalous" readings — those are often the most valuable.

The implementation is simple but the discipline is hard. Every time I write validation logic, I ask myself: "If this rule had existed last month, what real events would it have filtered out?" Usually I don't know the answer, which is exactly why I shouldn't be dropping data.

Anomaly detection needs context

Let's talk about where anomaly detection actually belongs, because I see this done wrong constantly.

The temptation is to put anomaly detection as early as possible in the pipeline. "We want to alert quickly! We need low latency!" So teams run anomaly detection on raw sensor streams, or on lightly cleaned data, and they wonder why the false positive rate is so high.

The problem is context. A temperature reading of 85°C from a sensor on machine 12 is completely normal during one shift and a critical warning during another. A pressure spike in isolation might be noise, but a pressure spike correlated with a temperature drop across multiple sensors is a clear pattern. Anomaly detection without context is just noise detection.

In my current architecture, anomaly detection happens after dimensional enrichment. By the time the model runs, each sensor reading has been joined with factory metadata, shift information, machine operating mode, and historical baselines for that specific machine under those specific conditions. The model isn't asking "is this reading unusual?" It's asking "is this reading unusual for this machine, during this shift, given what we know about normal operation?"

This requires more data movement and more compute. It adds latency. But the alternative is an alerting system that cries wolf so often that operators learn to ignore it. I'd rather have accurate alerts that arrive 30 seconds later than instant alerts that nobody trusts.

The one exception is obvious threshold violations. If a temperature sensor reads 500°C when the physical maximum for that sensor is 200°C, you don't need a sophisticated model to know something is wrong. Those checks happen early, flagged as data quality issues rather than operational anomalies. But the subtle stuff, the gradual degradation, the unusual patterns, the early warning signs, that needs the full context of enriched data.

Operations and Maintenance need different views

Something I wish someone had told me when I started building industrial dashboards: operations teams and maintenance teams need completely different views of the same data.

Operations wants to know what's happening right now. They need a dashboard that refreshes every minute or two, shows current status at a glance, and highlights any machine that needs immediate attention. The design principle is minimize cognitive load. A quick glance should tell them whether to worry.

Maintenance wants to know what's going to happen. They need historical trends, degradation curves, predictions about when components will need replacement. They don't care about minute-by-minute updates; they care about weekly and monthly patterns. The design principle is maximize insight. Give them the data to make proactive decisions.

I used to try to build one dashboard that served both. It was a disaster. Operations found it too cluttered with historical charts they didn't need. Maintenance found it too focused on real-time metrics that weren't useful for planning. Both teams complained that they couldn't find what they needed.

Now I build separate consumption endpoints, each optimized for its specific use case. The production dashboard is wide tables, pre-aggregated, heavily cached, designed for fast reads. The predictive maintenance reports are deeper, more detailed, with drill-down capabilities and longer historical windows. They're fed by different aggregations of the same underlying data, but they're shaped completely differently.

The third consumption pattern I've learned to build explicitly is real-time alerting. This isn't a dashboard. It's a feed that goes directly into the plant's notification systems. When an anomaly flag goes true on a critical machine, operators get a push notification immediately. This has different requirements than either dashboard: it needs lower latency, higher reliability, and much stricter filtering to avoid alert fatigue.

Three consumption patterns, three different designs, one underlying data platform. Trying to serve all three from a single "gold table" is how you end up with something that works poorly for everyone.

Simple Beats Clever

The most counterintuitive lesson I've learned about industrial data: simple transformations are almost always better than clever ones.

Early in my career, I was proud of my sophisticated data quality rules. Statistical outlier detection! Machine learning for anomaly classification! Automated schema inference! The pipelines were elegant and impressive in architecture reviews. They were also impossible to debug when they broke, which was often.

These days, my transformation logic is boring. Parse timestamps using explicit format strings. Cast numeric values with fallback to null on failure. Check range limits against hardcoded thresholds from equipment specifications. Flag records that fail validation instead of dropping them. Every transformation can be explained in one sentence.

The sophistication moves to the end of the pipeline, where it belongs. The enrichment logic that adds factory context? Simple joins against dimension tables. The anomaly detection that catches equipment failures? A dedicated model with its own testing and monitoring, not embedded in a transformation job. The aggregations that feed dashboards? Explicit SQL that anyone can read and verify.

This approach makes debugging possible. When the plant manager calls at 5 AM asking why something went wrong, I can trace the data flow in minutes. I can show exactly what the source sent, what transformations were applied, what validation checks passed or failed, and how the final result was calculated. No mysteries, no black boxes, no "the model decided this was an outlier but I'm not sure why."

I want to close with something that has nothing to do with specific technologies or patterns, but has fundamentally changed how I work: I sketch every architecture before I build it.

Not as documentation. Documentation comes after. I mean as a design tool, a way of thinking through the data flows before writing any code. When I can see the whole system on a single page, sources on the left, consumption on the right, all the transformations in between, problems become obvious that would be invisible in code. "Wait, where does the factory dimension data come from?" "Why are there two paths from this source to this output?" "What happens when this source is unavailable?"

I use Datadef.io for this, though the tool matters less than the habit. The diagram I've been implicitly describing throughout this article — SCADA and PLC sources, cleaning and validation, factory enrichment, production dashboards and maintenance reports — I drew it before I wrote a single line of Spark. And I caught three significant design issues just by looking at the picture.

The diagram also becomes the onboarding artifact. When new team members join, I don't hand them thousands of lines of code. I show them the picture. "Data flows from left to right. These are our sources, these are their quirks. This is where cleaning happens. This is where we detect anomalies. These are the outputs and who consumes them." Ten minutes, and they have a mental model that makes everything else easier to understand.

Factory floor data is unforgiving. The sensors are unreliable, the volumes are massive, the latency requirements are real, and the cost of errors is measured in broken equipment and production downtime. I've learned to build systems that are simple enough to debug at 5 AM, flexible enough to preserve the signals I might not recognize as important yet, and explicit enough that I can explain every transformation to a skeptical plant manager.

Draw first. Build simple. Debug fast. The rest follows.

T.L

What's New

Start with Production-Ready Templates

Stop staring at blank canvases. We've built three comprehensive starter templates based on real-world data architectures:

Banking Data Platform — Complete financial infrastructure with payments processing, fraud detection, and compliance reporting. Everything from raw transactions to executive dashboards.

Marketplace Analytics — End-to-end two-sided marketplace intelligence. Track supply and demand, optimize matching algorithms, and monitor transaction health across your platform.

Low-Code Analytics Platform — Perfect for no-code tools and workflow platforms. Monitor app builder usage, feature adoption, and user activation flows.

Each template comes with dozens of pre-built nodes, realistic data flows, and comprehensive documentation. Click "Use Template" and you're minutes away from a professional diagram.

Drawing That Just Works

Remember the joy of using Draw.io? We've brought that same intuitive experience to data lineage diagrams.

Professional Shape Tools — Create arrows, rectangles, circles, and text boxes that look crisp and professional. Every element has consistent styling, smooth interactions, and the visual polish your stakeholders expect.

Dedicated Text Editing — No more cramped text boxes. Edit text in a dedicated sidebar with full typography controls. Choose fonts, adjust sizes, align text, and see changes in real-time.

Streamlined Interface — We removed the confusing accordion menus and put everything you need right at your fingertips. Color picking, sizing, and styling are now effortless.

Dashboard That Scales With Your Team

Your dashboard should help you work faster, not slower. The new experience puts your most important projects front and center.

Smart Recent Activity — See your last three projects with node counts, role indicators, and update timestamps. Jump back into any diagram with one click.

Better Project Cards — Clean, modern design that shows what matters: project details, your role, and recent activity. Everything loads fast and looks professional.

Team-Ready Onboarding — New users get clear guidance and quick access to templates. No more wondering "what should I build first?"

Built for Team Collaboration

Great diagrams are built by teams, not individuals.

Project Invitations — Invite teammates by email or username with specific roles (Viewer, Editor, Admin). They'll get a clean notification and can start collaborating immediately.

Public Sharing — Make any project publicly viewable with read-only access. Perfect for sharing templates, documentation, or showcasing your work.

Role-Based Access — Fine-grained permissions ensure the right people can edit while others can view and comment.

Why This Matters

Data lineage shouldn't require a PhD in data engineering. Whether you're documenting existing systems, planning new architectures, or onboarding team members, Datadef now gets you from idea to professional diagram in minutes, not hours.

The templates alone will save you weeks of work. The improved drawing experience means you'll actually enjoy creating diagrams. And the collaboration features ensure your whole team can contribute.

Get Started Today

1. Explore Templates — Browse our starter gallery and find one that matches your use case

2. Try the New Drawing Tools — Create shapes, add text, and see how intuitive professional diagramming can be

3. Invite Your Team — Share projects and collaborate in real-time with role-based access

Ready to transform how your team thinks about data? Dive into v0.1.2 and see the difference.

What's Coming Next

We're already working on the next wave of improvements:

• Advanced Template Customization — Fork templates and create your own organizational standards

• Enhanced Collaboration — Comments, suggestions, and real-time editing

• API Integration — Connect your diagrams to live data sources and monitoring tools

• Enterprise Features — SSO, advanced permissions, and audit trails

Community & Feedback

Love the new features? Have ideas for what's next? We'd love to hear from you:

• Share your diagrams on Twitter with #datadef

• Send feedback directly through the app

Test it out!

v0.1.2 is live for all users. Existing projects will automatically benefit from the improved dashboard and collaboration features. Ready to try the new templates and drawing tools?

Start Building →

Six months ago, if you'd told me that a mundane "Export Project" button would become the cornerstone of an entirely new way of thinking about data modeling, I would have laughed. Today, I'm watching myself spin up complex data lineage diagrams in under 30 seconds, and I'm still a little stunned by how we got here.

A geometry problem

Let me start with a fundamental truth about AI that took me way too long to internalize: AI doesn't see the world the way we do.

When you're building a visual tool like Datadef, where users drag nodes around a canvas, connect them with edges, and build these beautiful, sprawling maps of their data architecture, you naturally think in terms of spatial relationships. Node A goes here, Node B connects to it there, the whole thing needs to look balanced and readable.

But ask an LLM to generate a canvas layout directly, and you'll get something that looks like it was designed by someone wearing a blindfold. The models are incredible at understanding relationships, dependencies, and hierarchies, but they're terrible at translating that understanding into X/Y coordinates that actually make visual sense.

This became painfully obvious during my early experiments with AI-generated templates. I'd feed GPT-4 a description like "create a data pipeline for an e-commerce analytics system," and it would dutifully generate all the right components, customer data, order processing, inventory management, reporting layers, but the spatial arrangement would be completely nonsensical. Nodes overlapping, edges crossing at bizarre angles, the whole thing looking like a bowl of spaghetti.

Using simple JSON

The breakthrough came from an unexpected direction. Months earlier, I'd built what I thought was just a basic utility feature: the ability to export entire Datadef projects as JSON files. The use case seemed straightforward, users wanted to share their work, back it up, maybe import it into different environments.

The technical implementation was relatively simple. Take everything that makes up a project, nodes, edges, documentation, glossary terms, metadata, and, serialize it into a structured format:

{
  "nodes": [
    {
      "id": "customer_data",
      "type": "data_source",
      "position": {"x": 100, "y": 200},
      "properties": {
        "name": "Customer Database",
        "description": "Core customer information...",
        "schema": {...}
      }
    }
  ],
  "edges": [
    {
      "source": "customer_data",
      "target": "analytics_pipeline",
      "relationship_type": "feeds_into"
    }
  ],
  "documentation": {...},
  "glossary": {...}
}

What I didn't realize at the time was that I'd accidentally solved the AI geometry problem. JSON is a language that LLMs speak fluently. They can read it, understand its structure, learn patterns from it, and, most importantly, generate new, valid JSON that follows those same patterns.

Instead of asking AI to think spatially, I could ask it to think structurally. Instead of "place these nodes on a canvas," I could ask "generate a JSON representation of this data architecture." The visual layout could be handled separately, using algorithms that actually understand geometry.

The 30-Second miracle

Once this clicked, everything changed. I started feeding sample Datadef projects to GPT-5, watching it learn the patterns of how we represent different types of data architectures in JSON. Then I flipped it around: give it a natural language description, ask it to generate the JSON for a complete project.

The results were genuinely stunning. A user could type something like:

"I'm building a recommendation engine for a streaming platform. I need to track user viewing behavior, content metadata, generate embeddings for similarity matching, and feed recommendations back to the frontend."

And within seconds, they'd have a complete workspace: data ingestion nodes, processing pipelines, ML model components, API endpoints, all properly connected and documented. Not just the high-level architecture, but the detailed schemas, the relationship types, even suggested glossary terms for domain-specific concepts.

What used to take experienced data architects hours of careful planning and layout now happened in the time it takes to grab a coffee.

Beyond templates: The conversational workspace

But here's where it gets really interesting. Once you have AI that can generate complete project structures from natural language, you're not limited to just starter templates. You can have conversations with your workspace.

"Add a real-time fraud detection component to this payment processing pipeline."

"Reorganize this to follow a medallion architecture pattern."

"Generate documentation for all the nodes in the customer journey section."

Each request results in precise JSON modifications that the system can apply instantly. The workspace becomes fluid, responsive to natural language in a way that traditional GUI interactions never could be.

I've watched users iterate on complex data architectures by simply describing what they want to change, rather than clicking through dozens of property panels and relationship dialogs. It's like having a conversation with your diagram, and the diagram actually understands what you're saying.

JSON limitation and what's next

Now, let me be honest about where we are versus where we're heading. Right now, this conversational interaction is still pretty basic, it's essentially AI generating modified JSON blobs that get imported back into Datadef. It works, but it's a bit like having a brilliant architect who can only communicate through detailed blueprints rather than walking around the construction site with you.

The real future lies in proper API integration. What I'm excited about is implementing something like the Model Context Protocol (MCP) or building dedicated API endpoints that let AI interact directly with Datadef's internal systems. Instead of "generate new JSON and replace everything," we'd have granular operations like:

• addNode(type, position, properties)

• updateDocumentation(nodeId, content)

• createRelationship(source, target, type)

• reorganizeLayout(pattern, constraints)

This isn't just cleaner architecturally, it opens up entirely new possibilities. AI could make incremental changes while preserving user customizations, maintain undo/redo history properly, and even collaborate with multiple users in real-time. The conversational workspace stops being a party trick and becomes a genuine new interface paradigm.

Think about it: instead of learning complex software interfaces, users could just talk to their tools. The JSON approach proved the concept works, but proper API integration will make it feel natural.

Datadef usage : some compound effect

What's fascinating is how this creates a compounding effect, though not in the way you might initially think. Let me be clear upfront: Datadef doesn't access user project data. Your architectures, your business logic, your sensitive information, all of that stays completely private.

But here's what's interesting: the patterns of how people structure data architectures are surprisingly consistent across industries, even when the specific content is completely different. The AI learns these structural patterns from publicly available examples, documentation, and architectural best practices, not from user data.

When a user describes building "a real-time fraud detection system for payment processing," the AI draws from its understanding of common fraud detection patterns, typical payment processing flows, and industry-standard architectures.

The result is still remarkably contextual. A user working on healthcare data pipelines gets suggestions that naturally incorporate HIPAA compliance patterns, proper audit logging, and the data segregation approaches common in that space. Someone building financial analytics systems gets architectures that reflect standard risk management frameworks and regulatory reporting structures.

The AI doesn't just generate generic template, it generates contextually appropriate architectures that feel like they were designed by someone who actually understands your problem space, while keeping your actual implementation details completely private.

Accidental innovation : from a boring feature to exciting opportunities

This whole experience has shifted how I think about the relationship between AI and creative tools. We often imagine AI as this separate assistant that we consult for specific tasks. But what I'm seeing with Datadef is something more integrated: AI as a medium for creative expression, not just a helper.

When you can describe complex systems in natural language and have them materialize as structured, visual representations, you're operating at a different level of abstraction. You're thinking about the what and why of your architecture, and letting AI handle the how of representing it.

It's the difference between being a painter who has to mix their own pigments and one who can focus entirely on the artistic vision because all the tools just work.

The most important lesson here might be about how innovation actually happens in software. The features that transform user experiences aren't always the ones you plan for. Sometimes they emerge from the intersection of two seemingly unrelated decisions: choosing JSON as a serialization format, and experimenting with AI integration.

That boring export feature? It wasn't just about portability. It was accidentally creating a universal language that both humans and AI could speak fluently. And that changes everything.

Six months ago, I thought I was building a better diagramming tool. Today, I realize I've stumbled into something much more interesting: a new way of thinking about the relationship between human creativity and machine intelligence. And it all started with a simple JSON export button.

I will finish with a personal quote :

Sometimes the most powerful architectures are the ones you never meant to design .

In most companies, the architecture of project management instruments usually revolves around elements like tasks, milestones, and timelines. These tools, though very effective in delivering software products or handling general business operations, somehow lag in reflecting the specifics that make data projects different.

Data project management challenges

Compositions ban not only refer to tasks but also talk about flows. The fact is, data fades away from the mixed sources through several layers like transformation, modeling, and quality control before it is delivered to a business user. This way, the data gear "learns" definitions, assumptions, and ownership. So to "manage" a data project really means to manage the data lineage, the documentation for every piece of the data, and the governance structures for the compliance and trust assurance.

My impression is that many teams piece this together using different tools. Miro might show architecture at a very abstract level, Jira will be used to keep track of implementation tasks, Confluence or Notion will be utilized for storing definitions, and spreadsheets will be holding Indicators. Such fragmentation always leads to the same problem: The map of the organization is out of sync with the territory. The diagrams are not updated when pipelines change. The glossaries do not match the metrics that are in production. Documentation turns from a place where half-done pages coexist to a graveyard. Compliance inspections become detective work.

With Datadef.io, I am trying to bridge such a gulf. The objective is to connect the three - diagrams, glossaries, and documentation - instead of keeping them fundamentally different. A canvas is the core analogy: the teams can show the data they have in the way that matches the origin, changes, and destination. But it is not like a pure whiteboard - every object on the canvas is deep. Once a table, an indicator, or a connection is described, then it can be used without the need for further description throughout the glossary, documentation, and project inventory. In case of any change, the change will be communicated.

Document is the second level. Since each element of a diagram is associated with metadata, the documentation of the project, no matter if it is a wiki, a PDF, or a Word file, can be automatically generated. Here, the headache of "keeping documentation up to date" turns into a resolved issue: documentation is not a parallel activity anymore, it is simply a byproduct of the project work. The assistance of AI makes raw descriptions done by the team into user-friendly narrative documents cutting down the description time for stakeholders.

Eventually, governance is a part of the design rather than an afterthought. The features of ownership, quality, rules, examples, and compliance can be among those that are connected to objects. It is, therefore, that an audit or an ISO inspection can use the very same knowledge base that the team accesses for development.

Practical application and comparaison with other solutions

A common case can be a team working on a pipeline that is the data source for a Power BI dashboard. Instead of the team going through stages like describing the architecture in Miro, defining indicators in Excel, and writing a Word document for compliance, they can open the canvas and map the complete flow there, mark each element with definitions and responsibilities, and let the system make consistent documentation across all outputs. The organization acts upon the definitions that are the same for all and governance is no longer a property of the workflow at a later stage but occurs simultaneously with it.

Some people tend to rely on extremely complicated solutions like Collibra or Informatica which, albeit being very powerful, are, most of the time, too cumbersome for small teams. Others are using generalist tools like Confluence, Notion, Jira, and Miro, which are characterized by being quite agile but hardly synchronized or specially tailored for data lineage. An overwhelming majority of people if not all resorting to workarounds of combining spreadsheets, diagrams, and texts are there as well.

Conclusion

In short, the way most organizations handle data projects today is fragmented: architecture lives in one place, definitions in another, and documentation somewhere else entirely. This separation creates friction, slows down collaboration, and undermines trust in the data itself. What I am building with Datadef.io is an attempt to unify those layers—lineage, documentation, and governance—into a single, living workspace where change in one place reflects everywhere.

The project is still young, but I believe the foundation is solid enough to be useful already. If you have faced the same pain points—out-of-date diagrams, scattered glossaries, compliance reviews that feel like detective work—I would love for you to try it, challenge it, and tell me what works or doesn’t. Datadef is not meant to replace all your tools, but to give data teams a space designed specifically for the realities of managing complex data projects.

You can explore it here: Datadef.io.

TL;DR

• Lineage highlights across nodes, edges, and table columns for instant context

• Export/Import now includes indicators (with handle remapping), lineage relationships, wiki documents, and columnIndex

• Copy/Paste and Duplicate fixed for indicator edges with correct handle/ID remapping

• Cleaner image export (no selection borders or resize handles)

• Polished edge tips on highlighted edges

• Wiki: default “Project Documentation” title + stable title editing (no more UI resets)

Check all the changes at : datadef.io

Why this release matters

When you select a node in the Lineage panel, you should immediately feel how data flows. In v0.1.1, connected nodes and edges light up while everything else takes a back seat. For tables, even the relevant columns highlight. It’s a tiny interaction that changes how fast you can think with the canvas.

Project portability also took a big step forward. Export/Import now includes indicators (with per-node cloning and handle remapping), lineage relationships, wiki documents (with hierarchy and links), and columnIndex. You can move a project and trust it’ll look and behave the same.

And for writing: the Wiki now starts with a sensible “Project Documentation” title and your title edits stick—even when you’re in the notion-style editor body. Simple, but it removes friction.

What’s new

Lineage highlight system

• A shared highlight state in the canvas context

• Unified Lineage panel dispatches rich highlight metadata (nodes, edges, and column maps)

• Nodes and edges dim/highlight smoothly; TableNode supports column-level highlighting

• Table refactor: faster, cleaner column rendering and interaction to make column-level lineage practical and performant

Export/Import coverage

• Indicator edges: source/target handles remapped via index tokens

• Lineage relationships: exported/index-mapped and correctly re-linked on import

• Wiki documents: hierarchy (parentIndex), linked nodes (linkedNodeIndex), and metadata

• nodeIndicators.columnIndex persisted to keep column-level lineage fidelity across environments

UX polish

Clipboard + duplication

• Indicator-aware copy/paste/duplicate: indicators are cloned per-node; handles remapped; edges point to the right indicators

• Table copy/duplicate respects column associations so column-indexed indicators remain coherent after paste/duplicate

Image export

• Selection borders and resize handles are hidden during capture for cleaner images

Edge visuals

• The highlight glow overlay no longer renders a marker on top of the arrow. The tip now feels smaller and more precise without changing the underlying edge.

Wiki

• Default title “Project Documentation” on first creation

• Title stays put when editing the body in the notion-style editor

Fixes

• Multi-select duplication used to lose indicator edges. It now clones indicators and remaps edge handle IDs correctly.

• Export/Import used to miss parts of your project (indicator handles, lineage, wiki, columnIndex). It’s all in now.

• TableNode stability fixes: removed edge cases where column-bound indicators could desync during complex edits.

See you next week

Datadef v0.1.1 lays another brick in the foundation, we’re smoothing the edges now so we can build faster later. The goal isn’t just polish, but momentum: every release should make the canvas feel more alive and the workflow more natural. From here, it’s about scaling the vision, bigger projects, richer context, and more ways for Datadef to grow with you.

Check all the changes at : datadef.io