By Peter Zervas, Senior AI Solution Architect @ Cognitive Creators

At Cognitive Creators, our days are typically filled with coding, LLM solutions, and pushing the boundaries of AI. But occasionally, something lands on our desks that makes us sit up and take notice. That's precisely what happened when I was asked to test Runway ML, an AI-powered video generation tool.

In this post, I will take you on my journey with Runway ML - from my first awe-inspiring (and occasionally hilarious) attempts at AI video creation to our more ambitious experiments. Spoiler alert: it involves floating box lids and a shark that would make Spielberg proud.

The Beginning: From Code to Canvas

My adventure began on a typical Tuesday. I was knee-deep in templates for Claude 3.5 Sonnet agents when the call came to test Runway ML. As someone accustomed to cutting-edge AI's "wow" moments, I'll admit my expectations were sky-high. Could this tool really turn my words into Oscar-worthy short films?

My first task seemed simple: create a 500-character prompt for a Cognitive Creators advertisement. Little did I know that condensing an epic transformation sequence into 500 characters is like trying to fit an entire symphony into a ringtone. Challenge accepted!

With a prompt describing a figure in a dark room, a glowing box, and a forest with circuit-leaf trees, I hit 'Generate'. Ten seconds and one forest later (dark room? What dark room?), I found myself in that familiar space between appreciating current AI capabilities and dreaming of future potential.

It wasn't the mind-blowing transformation I'd envisioned, but it was intriguing. More importantly, it was just the beginning. Runway ML had more surprises in store, and I was ready for the ride.

So, grab your popcorn as we dive deeper into the world of AI-generated videos. We'll explore Runway ML's strengths and quirks and what they might mean for the future of content creation. Trust me, you won't want to miss the shark-infested tale coming up next...

From Sharks to Sentient Robots: Pushing Runway ML to its Limits

Remember that shark-infested tale I promised? Well, it all started when we heard about OpenAI's Sora creating a beach scene with a surprise shark appearance. Naturally, we had to see how Runway ML would handle a similar prompt.

Our version? Let's just say it won't be replacing Jaws anytime soon. While Sora's shark sported a cheesy grin and featured a beach-goer who seemed to have either broken their neck or become possessed (the jury's still out on that one), our Runway ML creation took a different approach to aquatic terror.

In our video 'Shark Surprise' video (displayed below), we witnessed a guy casually walking on water - because who needs physics, right? But the real star of the show was our shark. This wasn't your average great white; oh no. Our shark appeared to be made of what I can only describe as "mimetic polyalloy" - you know, that liquid metal stuff from Terminator. T-1000, eat your heart out!

The best part? Our beach-goers didn't mind one bit. Liquid metal shark? A guy walking on water? Just another day at the AI-generated beach, apparently.

It was a hilarious reminder that AI, for all its advancements, still has a wonderfully weird side. These unexpected interpretations aren't just glitches; they're windows into the bizarre, often amusing way AI "thinks."

Emboldened by our foray into beachside horror, I decided to up the ante. My following prompt was an ambitious tale of a tin robot toy discovering sentience:

> "Tin robot toy marches in an endless loop. Suddenly, its painted eyes flicker with life. Wind-up key spins wildly. Metal frame stretches, grows. Colours shift from primary to iridescent. Jerky movements become fluid. Ticking sound evolves into a low hum. Robot's head expands, translucent with visible neural networks. It stops, looks around curiously. Reaches out, touches a book. Absorbs all knowledge instantly. Turns to camera, smiles."

The result? Well, let's just say it's straight out of a techno-horror fever dream. The robot, missing an arm (budget cuts, perhaps?), lurches through its transformation with all the grace of a malfunctioning blender. It's a vivid, if slightly terrifying, metaphor for the current state of AI - powerful, full of potential, but still very much in its awkward teenage phase. (Brave souls can witness the uncanny valley incarnate. Don't say I didn't warn you!)

As I watched my one-armed robot lurch across the screen, I couldn't help but draw parallels to the evolution of language models. Both are on a journey from rigid, pre-programmed responses to something more fluid and adaptable. However, in the robot's case, "fluid" might be a generous description.

It was around this time that I realized something crucial—Runway ML doesn't do sound. All those carefully crafted audio cues in my prompts are wasted bytes. This is a humbling reminder to always read the manual—or at least the FAQ.

Pushing further, I tried a prompt closer to home for Cognitive Creators:

> "Slow pan across dictionary. Words glow, lift off pages. Camera follows as they form neural network in air. Zoom in: nodes are words, connections are context relationships. Graph pulses as AI processes language. New words appear, connections shifting rapidly. "Transformer", "GPT", "fine-tuning" materialise. Zoom out: network expands infinitely, simulating LLM's vast knowledge. Pull back: AI-generated text flows from network back to book, rewriting pages in real-time"

The result was... interesting. While it captured the essence of words floating and connecting, some of the text was, shall we say, creatively spelled. "FINE-TUNDING" made an appearance, as did "FRANTOWINING" and "KETOIT". I'm still unsure what they mean, but I'm considering adding them to our company jargon. (Witness the birth of a new language - linguists, take note!)

Despite these quirks - or perhaps because of them - my colleagues found the videos fascinating and amusing. The shark video remains our office favourite, perfectly encapsulating AI's current state: impressive, hilarious, and slightly off.

Reflecting on my Runway ML adventure, I realised it was far from my usual work with text-based AI and carefully controlled agents. The lack of fine-tuned control was both freeing and frustrating. After years of precision calligraphy, it felt like finger painting - messy, unpredictable, and undeniably fun.

In the end, while Runway ML might not be replacing Hollywood directors just yet, it's a powerful reminder of how far AI has come - and how far it still has to go. It's a tool that rewards creativity and a healthy sense of humour, perfect for when you need a one-armed robot or a shark with questionable anatomy.

Personal Reflections: The Surprising Journey of AI Creativity

My adventure with Runway ML has subtly shifted my perspective on AI's role in creative fields. While I've always been excited about AI's potential, seeing it bring imagination to life in video form was truly eye-opening. It's not just about automating tasks anymore; it's about empowering people to be more creative, to visualise ideas they might never have attempted before. The prospect of democratising video creation, making it accessible to those without years of technical expertise, is both thrilling and slightly daunting.

The most surprising and exciting moments came when Runway ML tackled landscapes. It's as if the AI has a particular affinity for natural beauty. One standout was a SORA comparison prompt:

> "Aerial view of Santorini during the blue hour, showcasing the stunning architecture of white Cycladic buildings with blue domes. The caldera views are breathtaking, and the lighting creates a beautiful, serene atmosphere."

Runway ML's interpretation was impressively close to the mark, capturing the essence of Santorini's iconic beauty. It was a moment that made me pause and appreciate how far AI has come in understanding and recreating complex visual concepts.

Another test prompt yielded equally impressive results:

> "Aerial view of Uluru at dusk, showcasing the majestic red monolith against a backdrop of the vast Outback."

The result was breathtaking, with the sacred site glowing crimson against a vibrant sky. The AI even managed to represent the surrounding desert landscape with surprising accuracy. The dusk lighting was particularly well-executed, capturing the magical atmosphere of Uluru at that time of day. However, it wasn't without its quirks - the water feature I had mentioned in the prompt ended up looking a bit off. But considering the complexity of the scene, the overall result was remarkably close to what one might see in a real aerial shot of Uluru at dusk. [

However, it's not all picture-perfect landscapes. Runway ML showed its more avant-garde side with this prompt:

> "A lone mechanical bird, its wings gleaming with circuitry, soars between transparent skyscrapers pulsing with data streams."

The result was... let's say, unexpected. Instead of a sleek, high-tech avian, we got something that looked more like abstract digital art. It's a reminder that AI, like any creative entity, sometimes interprets our ideas in ways we never anticipated.

Limitations and Future Horizons

While Runway ML impressed me in many ways, it wasn't without its frustrations. The character limit on prompts felt particularly restrictive. As someone used to crafting detailed scenarios for language models, I found myself constantly hitting the prompt ceiling, forcing me to prioritise and simplify my ideas.

I'd love to see future iterations offer more customisation options. Pre-built templates could be a game-changer, especially for users new to video creation. Even more exciting would be an AI-powered prompt improvement system. Imagine an AI that could take your rough idea and translate it into a prompt optimised for video generation, maybe even guiding you through a series of questions to refine your concept.

Despite these limitations, I'm incredibly excited about the future of this technology. We're in the early stages of a revolution in creative AI, and tools like Runway ML are just the beginning. As these systems evolve, I anticipate we'll see more intuitive interfaces, better understanding of complex prompts, and even more impressive outputs.

In the end, my Runway ML experiments left me with a mixture of awe and anticipation. It's clear that AI is not just learning to mimic human creativity, but to become a collaborative partner in the creative process. As we continue to push the boundaries of what's possible, I can't help but wonder: what wild and wonderful creations will we dream up next, with AI by our side?

Want to see more of our AI video adventures? Check out our YouTube channel www.youtube.com/@CognitiveCreatorsAI for other videos created using Runway ML and join us as we explore the frontiers of AI creativity!

‍