Michal Gancarski

I'll say it again → Analytics Engineers are the new champions of data modeling. Talking to them at last week’s dbt Labs meetup in Berlin made me double down with an even bolder claim: Analytics Engineers are the new champions of data modeling and **AI adoption**. Without governed data models, your AI-driven analytics just aren’t reliable. Thank you Eva Schreyer, Lucas Silbernagel, Tim Hiebenthal, Victoria Perez Mola, and everyone who came—it was such a wonderful experience!

83

12 Comments

What parts of the data model should be built in dbt? in the semantic layer? in the last-mile layer? The (many) posts about this usually make the same mistake. They only include the first two layers and skip the last-mile. That's like writing a recipe for a loaf of bread and skipping the baking instructions. Data people like to pretend like the last-mile doesn't exist. Because the last-mile often happens in excel or sheets, and our egos can't accept that. But the reality is our structured data pipelines have a trade-off: reduced flexibility. The end-users need their data in a format that can't be expressed by a structured pipeline. So (officially or unofficially) the last-mile happens. Let's break these down, honestly, without pretending the last-mile layer doesn't exist: 1) The transformation layer does the heavy lifting. It's a series of SQL steps for cleaning, combining, and normalizing data. The goal is a set of tables at the end of the pipe exposed for analytics. You should do as much as possible here because it's pre-computed (and the more you pre-compute, the faster your data tools will be). This is all done in SQL with a tool like dbt, Coalesce, or SQLMesh. 2) The semantic layer happens at query-time. This is where you'll dynamically slice, aggregate, and filter data - anything that would create redundant tables in the transform layer. For example, the semantic layer aggregates an orders table into daily/weekly/monthly sales (if you did this in the transformation layer, you'd need to point your users at 3 redundant tables). The semantic layer is built inside the BI tool, or using a standalone semantic layer such as Cube or Metricflow. 3) The last-mile layer is the one everyone neglects. The last-mile includes all steps after the query is run. Taking the big, tall table prepared by the semantic layer, and turning it into the right shape for your project. That includes things like pivoting, formatting, or certain additional calculations (percentage growth is the classic example). Sometimes you'll combine another query. This is all done in excel (the horror!!), or as table calcs/merged results in the BI tool. Each layer has its purpose. The transform layer prepares a clean, consistent dataset. The semantic layer handles aggregation and filtering. And the last-mile layer gets data into the desired format. As a general rule, you should push everything as far left as you can, but not farther. Why am I writing this now? Because we released our great new merged results feature in Zenlytic (link below). This is an important one for me since I've personally spent more time as a last-mile analyst than anything else. That's why the last-mile will always be a first-class citizen in Zenlytic.

23

6 Comments

Two days ago, I said OpenAI clearly had something up its sleeve for paid users. Today, they announced interactive tables and charts from live files—and it looks awesome. Data Analyser is one of ChatGPT's lesser-talked-about features because it had a somewhat rocky start. First of all, LLMs (Large Language Models) aren't great at anything related to math. This makes working with data... a problem. OpenAI circumvented this by getting ChatGPT to write a program (in code, which it's great at) and then using the code it wrote to analyse the data. This was a clever workaround, and it significantly improved the accuracy of its solution when dealing with data. But it was still somewhat clunky and sometimes took 2-3 tries to get an analysis right. Over time, GPT-4 improved its reasoning, which in turn improved its analysis, and Data Analyser has now become a tool I use almost daily to interrogate various data sets. Today's announcement, however, takes it over the utility threshold for *most* people. Building charts for presentations, understanding data sets from various angles and presenting stories with data is something many *want* to do, but few get to. That's about to change. For everyone who keeps saying, "LLMs are no longer progressing enough; we've hit a wall," this is another example of how we can make extreme productivity jumps without any underlying LLM improvement. There are 1000 more like this - some will come from OpenAI directly, others from the industry building on top of these models. The next few years are going to be interesting :) #FutureOfWork #ArtificialIntelligence #Productivity

60

14 Comments

Meta released "Self-taught evaluator" that evaluates generative systems and also improves itself when needed: 1. Reduces project timeline by ~70% 2. Reduces cost by ~50% 3. Automates ~90% of the most tedious task

10

1 Comment

One of the most annoying issues with graph databases? Hard to use with most BI tools, and hard to federate with other datastores. It's like "yeah, I totally DO want to be able to use some graph queries, but ultimately I need to surface their results often not in some funny graph-looking thing, but in a ranked table, or a bar graph... ya know, like most all data consumers look for" It's a silly but real roadblock - a consequence of a technology and query language that inherently is not rows and columns. But... like... a lot of graph queries.. the results can somewhat easily be LOGICALLY flattened, now how to use it in other tools? NeuronSphere uses a graph-database as the core of it's transformation control engine, and in order to easily view results from that graph along with other business and operational data, we developed a #trino plugin that allows mixing gremlin queries into your SQL, which means all the BI tools that can easily access #trino can now easily access a Neptune or Janus graphdb, including joins with non-graph sources. Being able to author and surface your graph queries directly in #superset via trino elevates your graph database to the equivalent usability as your rdbms, as all your jdbc/dbapi/etc tooling works seamlessly :) #meatbasedengineer

49

12 Comments

It’s time to standardize definitions for Open Source AI. Meta’s Llama-3, TII’s Falcon, Databricks Dolly, Mistral's Mixtral, Google’s Gemma are all described as Open Source. But they are all different and many don’t meet my (or most reasonable) definitions of Open-Source models. Let’s look at Llama-3 for example. You do get the weights of the trained model and can download the model and run it wherever you want. But you don’t get the source code for the model nor the training data. So you cannot recreate the model, contribute to its development, or fork it easily in different directions (not counting Finetuning which is a capability available with closed source models as well). Further, there exist several licensing restrictions as well (e.g. if you have over 700M MAUs). To me, that is not an Open Source model. It is an Open-Weight model. In contrast, Dolly from Databricks also claims to be Open Source. But it comes with source code, training data, and model weights. And that fits most definitions of Open Source. This is not to suggest Open Weight models don't have value in the market. I love the greater control and data security that comes from running your own models within your cloud bucket/container. Open Weights do offer some level of transparency as well (though minimal relative to open source). But calling them Open Source betrays the philosophy behind the movement. PS: Open versus closed source should not be mistaken with open vs closed platform strategy (e.g. MS Windows and Android follow an open strategy versus Mac OS is a closed OS). Microsoft Windows is a closed source OS. But any hardware manufacturer (like Dell) can install it on their device. So their distribution strategy is open but the OS itself is closed source. Android, on the other hand, is both an open platform (e.g. available on Samsung) and open source (though Google exercises version control). In that sense, Llama is a far more open model because anyone can download and use the model but that's not what open source is all about. More discussions on AI in my newsletter at https://lnkd.in/gZ9cCMfX

94

8 Comments

🤔 "Should we build or buy our data infrastructure?" After making this decision dozens of times at companies like Google, LinkedIn, and various startups, here's my framework: Buy when: The problem is well-understood Solutions are mature Your needs match industry standards Engineering time is better spent elsewhere Build when: You have unique requirements The solution is core to your competitive advantage Existing solutions don't scale to your needs You need deep customization Most important lesson: The true cost of "build" isn't the initial development - it's the years of maintenance ahead. What's your experience with build vs buy decisions?

12

1 Comment

🧠 𝐀𝐫𝐞 𝐘𝐨𝐮 𝐀𝐜𝐜𝐢𝐝𝐞𝐧𝐭𝐚𝐥𝐥𝐲 𝐒𝐡𝐚𝐫𝐢𝐧𝐠 𝐓𝐨𝐨 𝐌𝐮𝐜𝐡 𝐰𝐢𝐭𝐡 𝐆𝐏𝐓 𝐓𝐨𝐨𝐥𝐬? 🧠 We’ve all been there. You’re typing away into your favorite GPT-powered tool, asking for help with that tricky project, and BAM! You realize you’ve just shared a little too much information. 😳 But wait—what does it really mean to share your data with GPT tools? 🚦 𝐋𝐞𝐭’𝐬 𝐛𝐫𝐞𝐚𝐤 𝐢𝐭 𝐝𝐨𝐰𝐧: When you interact with GPT tools (like ChatGPT), you’re feeding it data—questions, problems, and sometimes, without realizing it, sensitive information! Now, GPT doesn’t save this data for itself (phew), but once it’s out there, it's out of your control. Here’s where things can get a bit dicey! 🎯 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐌𝐢𝐬𝐭𝐚𝐤𝐞𝐬 𝐖𝐞 𝐌𝐚𝐤𝐞 (𝐖𝐢𝐭𝐡𝐨𝐮𝐭 𝐄𝐯𝐞𝐧 𝐑𝐞𝐚𝐥𝐢𝐳𝐢𝐧𝐠): "𝐂𝐚𝐧 𝐲𝐨𝐮 𝐡𝐞𝐥𝐩 𝐦𝐞 𝐟𝐢𝐱 𝐭𝐡𝐢𝐬 𝐜𝐨𝐧𝐭𝐫𝐚𝐜𝐭?" 📄 Sure, it’s nice to have GPT look over your contract wording, but suddenly you’ve shared client names, prices, and maybe even confidential terms. Yikes! "𝐈 𝐧𝐞𝐞𝐝 𝐭𝐨 𝐮𝐩𝐝𝐚𝐭𝐞 𝐨𝐮𝐫 𝐬𝐚𝐥𝐞𝐬 𝐩𝐢𝐭𝐜𝐡." 🎯 Whoops! You’ve just casually dropped your company’s strategy, pricing model, or that new product launch you're supposed to keep under wraps. "𝐖𝐡𝐚𝐭 𝐝𝐨 𝐲𝐨𝐮 𝐭𝐡𝐢𝐧𝐤 𝐚𝐛𝐨𝐮𝐭 𝐭𝐡𝐢𝐬 𝐜𝐨𝐝𝐞?" 💻 Asking for help with some code is great... until you realize it contains proprietary algorithms, login credentials, or access keys. Oops! "𝐂𝐚𝐧 𝐲𝐨𝐮 𝐰𝐫𝐢𝐭𝐞 𝐚 𝐋𝐢𝐧𝐤𝐞𝐝𝐈𝐧 𝐩𝐨𝐬𝐭 𝐟𝐨𝐫 𝐨𝐮𝐫 𝐧𝐞𝐰 𝐩𝐫𝐨𝐝𝐮𝐜𝐭?" 🚀 GPT nails the copy, but you also shared sensitive launch details or internal pricing discussions you weren’t supposed to leak. Double oops. 😬 𝐇𝐨𝐰 𝐃𝐨 𝐖𝐞 𝐀𝐯𝐨𝐢𝐝 𝐓𝐡𝐞𝐬𝐞 𝐎𝐨𝐩𝐬 𝐌𝐨𝐦𝐞𝐧𝐭𝐬? A few pointers below that might help you next time before you start typing away 𝐏𝐚𝐮𝐬𝐞 & 𝐓𝐡𝐢𝐧𝐤: Before you type, ask yourself, Would I want this information shared publicly? 𝐔𝐬𝐞 𝐃𝐮𝐦𝐦𝐲 𝐃𝐚𝐭𝐚: If you're seeking help on sensitive topics, make sure to use placeholder data instead of the real deal. 𝐁𝐞 𝐀𝐰𝐚𝐫𝐞 𝐨𝐟 𝐂𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐭𝐢𝐚𝐥𝐢𝐭𝐲: For business or sensitive conversations, GPT tools aren’t the place for full disclosure. There are NDAs, and then there’s GPT—know the difference. 😉 💡 𝐅𝐢𝐧𝐚𝐥 𝐓𝐡𝐨𝐮𝐠𝐡𝐭: GPT is an awesome productivity booster, but it’s only as safe as you make it! Let’s be smart about how we use it and avoid oversharing. Keep it fun, keep it safe, and happy prompting! 👇 Have you ever caught yourself oversharing with an AI tool? Let’s swap stories (the non-sensitive kind, of course!) in the comments! And if you found this helpful, give it a 👍 and share it with your network to keep the conversation going about AI safety! #GPT #AIAwareness #DataSafety #ProductivityTools #DataSecurity

30

1 Comment

80% of data projects fail 80% of AI projects fail 80% of Gen AI projects fail 80% of AI prototypes don't make it to production we love these stats in the data community but at what point do we data 'experts' start to question the validity of the throw away 80% xy and z? #datadriven #dataleaders #datatransformation #transformation #isthereajohnintheaudience

42

30 Comments

We overhauled the dashboard experience in Causal and I am loving the new look 🤩 The new layout is way more flexible and configurable to let users decide the best way they want to showcase their data. In the new dashboard layout: 🎯 Users can add new visuals directly in the position you want them (no more dragging from the bottom of the dashboard!) 💬 Text is more native, allowing you to break up your dashboard into digestible sections, and tell the story you want alongside your data 3️⃣ You can now have up to three visuals on one row (vs. two before), which will be especially useful for charts, KPIs, or side-by-side text blocks underneath visuals. 〰️ With two visuals on a row, you can now have a one-third: two-thirds layout (instead of just half & half), giving you more room for visuals that need space to breathe 🐭 Changing the position of visuals with drag and drop should feel a lot smoother + more responsive.

71

10 Comments

𝐒𝐩𝐞𝐧𝐝𝐚𝐭𝐚: A True 𝐄𝐧𝐭𝐞𝐫𝐩𝐫𝐢𝐬𝐞 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬 Solution (𝘐𝘵'𝘴 𝘯𝘰𝘵 𝘺𝘰𝘶𝘳 𝘨𝘳𝘢𝘯𝘥𝘱𝘢'𝘴 𝘴𝘱𝘦𝘯𝘥 𝘢𝘯𝘢𝘭𝘺𝘴𝘪𝘴 𝘢𝘯𝘺𝘮𝘰𝘳𝘦!) No "data warehouse", "data lake", or "data lakehouses" needed either! Nor do you need a star schema to avoid suffering in computational limbo either! (But if you still think PowerBI is a great presentation tool, it can build you a perfect star schema on millions of records in seconds ... ) Just real spend analysis, which includes the ability to: * extend the schema at any time * add new derived dimensions at any time * change those mappings at any time * build derivations, data views, and mappings that are dependent on other derivations, mappings, views, inputs, linked datasets, and so on, with real-time “recalculation” * create new views and reports WITHOUT dumping the data to Excel * preserve ALL of the above on cube data refreshes (that's right! you DON'T have to rebuild the cube from scratch on every refresh!) * do so in your OWN private copy of the cube (that inherits from the shared master cube) so you don't have to change the common cube for your own custom analysis and reporting needs * get the answer you need RIGHT NOW (not on the next refresh next month when you've forgotten why you were told to get the answer in the first place) So what else can it do besides every spend analysis report ever invented under the sun? * 𝐒𝐚𝐥𝐞𝐬 𝐃𝐢𝐬𝐜𝐨𝐮𝐧𝐭 𝐕𝐚𝐫𝐢𝐚𝐭𝐢𝐨𝐧 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬 Over Time By Salesperson … and Client Type ... that dynamically updates in real time on every parameter or "what-if" change * Real 𝐏𝐨𝐬𝐭-𝐌𝐞𝐫𝐠𝐞𝐫 𝐖𝐡𝐢𝐭𝐞-𝐒𝐩𝐚𝐜𝐞 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬 (if you think you can do this now, think again) * 𝐌𝐮𝐥𝐭𝐢-𝐯𝐚𝐫𝐢𝐚𝐭𝐞 𝐁𝐢𝐥𝐥 𝐑𝐚𝐭𝐞 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬 ... and more! https://lnkd.in/eM9Y4fez

9

Speeding up index creation Let's talk about speeding up index creation in PostgreSQL. If you've ever had the pleasure of waiting for hours while your database churns away at creating indexes on a large dataset, you know the feeling. Fortunately, Hans-Jürgen Schönig has some insights that may help you cut that time significantly. Indexes are crucial for optimal query performance. But creating them can feel like watching paint dry. Thankfully, there are techniques floating around that can make the process a bit less torturous. Speaking of insights, PGConf.dev 2024 had a lineup of talks worth tuning into if you missed it. Topics ranged from autovacuum mishaps to scaling databases at petabyte levels. Definitely a treasure trove for any PostgreSQL enthusiast. Also, if you fancy learning some magic tricks for psql, Elizabeth Christensen shares some nifty settings and presets that can make your life easier. So, whether you're indexing or attending conferences, the PostgreSQL community is here to keep you entertained and informed. Cheers to faster queries and smarter database management! Source: https://lnkd.in/e8SSrvJd

2

I don’t want certain data teams to use our BI platform. Not because I have anything against them (obviously), but because they just don’t need it. Smaller teams in smaller companies often don’t need: → A semantic layer → Self-serve abilities → Permissions functionality And for them to pay us and not get the right utility out of Zenlytic would be silly. For this example, a tool like Redash would make a lot of sense. It’s dead simple and works well for lean engineering teams with different needs than Zenlytic’s customers. I’m trying to highlight: The tools that make sense for your company might look a lot different than the ones that make sense for other teams. A small, engineering-led team doesn’t need the same data tooling as a team with self-service needs. Purchase accordingly.

18

6 Comments

Do you think that metric trees are a very nerdy topic in data? Well, then meet the super nerdy cousin of growth models. Reading Paul Levchuk's post last week about growth models triggered the need to build one again. Yes, I know that is weird. But it is really fun. What is different with growth models, and why is it so much fun? When I start with a metric tree, I will take this structure and extend it with more details about different growth initiatives. Like in a metric tree I might only have a metric "Visits from Substack". In a growth model I look at the whole funnel/journey/loop from a Substack perspective. What is the production function (Substack post published * (Avg. organic views + LI views + BS views) * View growth rate * Click rate = Visits from Substack). How does the conversion look different on this path. Where my current metric tree for my content work in 2025 has around 30 metrics, the growth model has over 200. But is this manageable or useful? Yes, it is. First of all, these metrics are all related and contextual. Therefore, understanding the impact between them and the big picture is quite straightforward. Second, they are great indicators to understand your model. Which initiative is driving growth by volume, which has good rates but is missing volumes? A great place to adjust your strategy.

54

15 Comments

C-suites struggle to see the value in data teams. Data teams struggle to communicate their value. Here's my method for fixing this (even if you're introverted): Nobody really looks over your shoulder to see who's doing what. Data is not unique in this, it's true of every job. The downside? The question of the value you bring to the table is more likely to come up – especially because all of your work is internal. You aren't selling the product or making the product so you're value isn't obvious. So you really need to sell yourself. To make matters worse? Generally, the only things that make it to the C Suite are the worst problems that need fixing. 95% of problems get solved before it gets to them. So the easiest way for a data person to prove their value? By putting the "data" in data person, backing up internal feelings with data. Example: • Your C Suite is worried about churn • They feel it's because of product bugs. • If you have data supporting otherwise, show it to them. Help them find the real root cause. • If you have data supporting their conclusion, go a layer deeper. Help them find what the company can change to fix the issue. You can present it internally, or record a loom (with a strong tl;dr written message). This helps you solve real problems your C-suite is worried about. That is how you showcase your value. Things like this are simple, but get you seen, heard, and valued. tl;dr: If you're a data person trying to prove their worth to the C Suite, give them data they didn't have to back up and go deeper in their hunches.

30

16 Comments

What squirrels have to do with Knowledge Graphs? 🧐 ⛔️ Nope, most likely they do not know that much about Databases, RDF, SQL, SPARQL, Java or Python. To expect that from squirrels would be real nuts. 🤯 💡 What squirrels know is how to build their own tiny world models that are dynamic, understandable to other squirrels and interconnect different domains or aspects of those worlds. 🐿️ Squirrels are "scatter hoarders," which means pretty much what it sounds like — they hoard their food, and scatter it in locations where they can easily access it. That's usually close to the tree holding their nest or den, but they often expand into areas of 7 acres (2.8 hectares). That suggests some pretty advanced memory skills. 🧠 To be able to do this squirrels need some geospatial memory that allows them to remember places where their nuts are stored and distances between those places. This must also include some memory about elevation. ⛰️ Moreover, they are able to connect places to the information about what is stored in those places - their different energy sources. 🔋 In other words, they memorise the terrain and different places in which they hoard in a way that does not make them run out of energy while climbing a hill before they find a nut in place where it is exactly supposed to be. 🤩 This means that they interconnect geospatial aspect with energy source aspect within the model. 💡 Their geospatial-food models are dynamic in a way that changes in the models are accommodated into those models immediately. Squirrels do not go to the place where they extracted their food from before. They remember it is no longer there. 👀 To build such a hoard is not an easy task. There may be other more lazy squirrels around snooping on their activities and hoping for an easy lunch. To keep bad neighbours away from the hoard, squirrels use few tactics. To make them confused, after digging some of the holes, they only pretend to burry the nuts. 😵‍💫 Sometimes they also crack nuts before burying them in order to give other squirrels impression of the food being not edible when it is found there. 🤢 This means that those tiny world models are understandable to other squirrels. Otherwise there would be no point in taking those steps while preparing a hoard. 💡 🤖 Similarly to that, Dynamic Geospatial Knowledge Graphs allow computer systems to store geospatial information about terrain, elevation, distance as well as interconnect it with other domains, like energy. This information is standardised and understandable to other computer systems or humans equipped with appropriate domain knowledge. 📚 If you'd like to learn more about this, go here: https://lnkd.in/gc3U5ruD 📚 If you'd like to lear more about advanced geospatial memory skills of squirrels: https://lnkd.in/gvFpuukT #knowledgegraphs #ai

27

24 Comments

Making Sense of Data Context: A Simple Guide We often see LinkedIn posts and read about "understanding data context" but what does this really mean in practice? For me, it boils down to asking these three questions in a respectful, safe environment where no one feels offended: • Inquire about data sources: • Would you mind sharing the sources of this data? • Once you know the sources, ask: • Who can tell me which systems touched this data along along the way? • Before delivering the dashboards or reports ask: • Why does it matter to our users? Pro tip: Ask these questions early in your data projects. They'll save you time, build trust with stakeholders, and ensure your analysis actually drives value. Remember: Better context leads to better decisions but make sure the environment is safe before you shoot your questions. What other things you do to learn the data context better? Please share. #businessIntelligence #dataContext #databestpractices #dataleadership #bi #dataengineering

6

💡 The Hidden Cost of "Easy" Data Platforms Fivetran no longer free to run dbt core models. Not surprising, but a reminder: What Usually Happens: - Start with managed services - Get comfortable - Your Data grows - Bills explode - Migration becomes painful Your data platform should be an asset, not a liability. #DataEngineering #StartupTech #DataPlatform

63

15 Comments

Is congestion pricing working? Let's take a look at the data in real time. The graph will show you zones compared to the day of the week before, and after congestion pricing went into effect.

1