Snowflake Cortex and the Future of Visual Intelligence: Multi-Modal AI in Action

Artificial intelligence is no longer limited to text-based input. With Snowflake Cortex’s powerful multi-modal capabilities , we can now generate rich insights from visual assets — including charts, marketing creatives, and even stock market trends — entirely within SQL using the Snowflake Data Cloud.

In this post, I walk you through three practical examples demonstrating how Snowflake Cortex can extract, compare, and analyze visual content without needing to export your data or switch tools. Let’s dive in.

Example 1: Named Entity Recognition (NER)

We begin by analyzing a visual — specifically, a cars image potentially somewhere in Japan/China. We ask the claude model about the number of cars & of what colors. Image is sourced from internet.

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    'What are the number of cars. If any, of how many colors',
    TO_FILE('@image_stage', 'cars.png'));

The model response is as follows -

In this image, there are 2 Honda Civic cars:
1. A yellow Civic
2. A blue Civic with racing/modification decals

So there are 2 cars in 2 different colors (yellow and blue) parked in what appears to be a Japanese shopping street or district (likely in an area like Osaka’s Dōtonbori or a similar entertainment district).

As can be seen from the response, the model correctly identifies the 2 cars, their colors, models & background scene from Japan’s Osaka.

Test case : Passed

What if we change the prompt to identify the entities alongwith their counts?

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    'Identify all the entities in the image alongwith their counts',
    TO_FILE('@image_stage', 'cars.png'));

This time the response is as follows —

In this image from what appears to be a Japanese street (likely in an area like Osaka’s Shinsekai), I can identify:

1. Cars (2):
- 1 yellow Honda Civic
- 1 blue modified Honda Civic

2. Japanese Signs/Banners (Multiple):
- Various illuminated shop signs with Japanese text
- Hanging banners and decorations

3. Buildings (Multiple):
- Several commercial buildings/shops on both sides of the street

4. Street Decorations:
- 1 large spherical lantern/decoration (appears to be a fugu/pufferfish-styled lantern)
- Multiple overhead banners and lights

5. Street Features:
- 1 main pedestrian walkway/street
- Patterned pavement/tiles

6. Lighting:
- Multiple shop lights
- Street lighting
- Decorative lighting installations

The scene is captured during evening/night time, showing a typical Japanese entertainment or shopping district atmosphere.

Test case : Passed

Example 2: Summarizing a Pie Chart with One Prompt

We continue with analyzing a visual — specifically, a pie chart showing the employment breakdown in mathematics-heavy careers. The image is sourced from snowflake docs

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
  'Summarize the insights from this pie chart in 100 words',
  TO_FILE('@image_stage', 'science-employment-slide.jpeg'));

It correctly analyses the image and provides following insights

“The pie chart shows the distribution of occupations where mathematics is considered “extremely important” in 2023. Data Scientists dominate with 48.7% , and Operations Research Analysts account for 29.6% . Statisticians and Actuaries make up around 15% combined, with traditional Mathematicians only comprising 0.6% . This visual underscores the industry’s tilt toward data-centric roles over traditional academic pathways.”

Test case : Passed

💡 Takeaway: In just one line of SQL, we’ve turned raw visuals into crisp, policy-relevant insight.

Example 3: Root Cause Analysis of a Stock Chart Crash

Images of financial charts often hold complex narratives. Snowflake Cortex enables visual insight generation that captures context, history, and consequences from a simple chart upload.

Here is the Boeing stock chart between Dec 2023 & March 2024 from yahoo finance

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
  'What led to the fall in boeing share price between Dec 2023 and March 2024. Include all market-related, company-related, competition-related, security-related data.',
  TO_FILE('@image_stage', 'boeing.png'));

It correctly analyses the image and provides following elaborate insights.

“The significant decline in Boeing’s share price between December 2023 and March 2024 was primarily driven by several major events and concerns:

1. Alaska Airlines Incident (January 2024):
- A Boeing 737 MAX 9 experienced a door plug blowout during flight
- Led to temporary grounding of 171 737 MAX 9 aircraft
- Sparked renewed safety concerns about Boeing’s quality control

2. Manufacturing and Quality Control Issues:
- FAA investigation revealed significant quality control problems
- Boeing was required to halt planned production increases of 737 MAX
- Multiple airlines reported loose bolts and other defects in their Boeing aircraft

3. Leadership Changes:
- CEO Dave Calhoun announced he would step down by end of 2024
- Board chairman Larry Kellner and head of commercial planes Stan Deal also departed
- Created uncertainty about future leadership and direction

4. Regulatory Scrutiny:
- Increased FAA oversight of Boeing’s production processes
- Required improvements in quality control systems
- Limited production expansion plans

5. Market Impact:
- Stock dropped from around $260 to approximately $180
- Loss of customer confidence
- Increased orders for competitor Airbus
- Potential financial impact from compensation to airlines

6. Business Implications:
- Delivery delays to customers
- Increased costs for inspections and fixes
- Potential loss of market share
- Damage to company reputation

These factors combined created significant investor concern about Boeing’s short and long-term prospects, leading to the substantial decline in share price during this period”

Test case : Passed

💡 Takeaway: What previously required comprehensive research and sentiment analysis across multiple datasets is now distilled from one static image using just one SQL function.

Example 4: Audience Comparison of Two Display Ads

Marketers often run A/B tests, but summarizing creative differences and identifying the personas can be subjective and time-consuming. Here’s how Snowflake Cortex handles it. Images are sourced from Snowflake documentation

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    PROMPT('Compare this image {0} to this image {1} and describe the ideal audience for each in two concise bullets no longer than 10 words',
    TO_FILE('@image_stage', 'creative1.png'),
    TO_FILE('@image_stage', 'creative2.png')
));

Cortex analyses both the creatives and suggests the right audience for each

“Image 1 “DISCOVER A NEW ENERGY”:

• Traditional SUV buyers seeking modern design with familiar functionality
• Pragmatic drivers who want gradual transition to newer technology

Image 2 “ELECTRIFY YOUR DRIVE”:

• Tech-savvy early adopters ready to embrace full electric vehicles
• Environmentally conscious drivers wanting a bold, progressive statement”

Test case : Passed

💡 Takeaway: Cortex not only compares tone and themes across visuals, but also nails down detailed micro-targeting strategies directly as SQL output — no manual creative brief required.

Final Thoughts

Multi-modal intelligence in analytics isn’t just a buzzword — it’s a paradigm shift. Whether you’re a data analyst, marketer, or executive, Snowflake Cortex allows you to interact with visual data naturally , intelligently, and scalably — all without leaving your secure Snowflake environment.

So next time someone sends you a chart or creative asset, don’t just view it — query it .

if you liked my post, consider giving it a few claps. You may also follow me on Linkedin.