New Features in Beta 3

Keita Mitsuhashi

Co-founder & COO

Morph Beta 3 has been released, available to invited users. This article introduces the updates in Beta 3.

Introduction of the File System

Providing an environment where SQL and Python can be freely combined to handle data has been a significant challenge in terms of realizing the flexibility of the Morph. Up to Beta 2 the design was similar to a low-code tool, requiring users to write code in specified locations according to set rules.

This approach had two main issues.

Firstly, it prevented users from bringing in their existing SQL or Python knowledge and best practices. Because low-code tools enforce rules, many programmers and tech-savvy business professionals cannot apply what they know in the code editor directly.

Secondly, from a product perspective, it always required the development of new features to meet new user needs. Developing new features entails designing databases, developing backends, and developing frontends, which necessitates a very long development period.

To address this, we redesigned Morph’s functionality as a framework and introduced a file system similar to that of a local computer into the product.

This allows users to obtain one of the true virtues of programming: limitless possibilities and ultimate flexibility.

The updates related to the file system include:

• Users can import static files of any format. Previously, only CSV files were allowed.
• Data transformations, charts, and reports are output as static files, mainly HTML files or CSVs.
• The publication of output is managed as access restrictions to the generated static files.

Managing the Canvas with Code

The Canvas, which is a core feature of Morph, is also managed internally with code.

In Morph Beta 3, the data canvas is executed by referencing a YAML file called morph.yaml. Changes made on the data canvas are reflected in each script file and morph.yaml.

This improves the stability of the data canvas with complex features and supports long-term operation.

Future Prospects

Reading through the updates so far, you might have an idea of what we are aiming for.

Morph has been redesigned as a programming framework that can operate from the command line. Within Morph, various functions are realized by executing the framework.

Over the next few months, we plan to open source parts of the framework, making it possible to utilize Morph's features in a local development environment.

Updates to Morph AI

Morph Beta 3 also brings significant updates to Morph AI. Morph AI is always accessible from the screen as a user's colleague and offers different modes depending on the file or page the user is viewing.

The following modes are currently implemented:

Building Data Pipelines

Morph AI handles the entire process from designing, executing, and evaluating the results of data pipelines. It not only retrieves aggregated results but also records the aggregation process on the Canvas, allowing the user to not only get results but also for reusing the constructed pipelines.

Upon receiving tables to be used and prompts in natural language from users, the following processes are automatically executed:

1. Planning: Upon receiving the user's request, it plans to construct the desired data pipeline. Even if multiple SQL and Python processes are needed, it plans the steps in stages. If there are discrepancies between the requested goal and the plan, it can also request corrections to the plan.
2. Coding and Execution: Based on the plan, it implements SQL and Python. Understanding the data schema allows Morph AI to implement code optimized for Morph's framework. This code is immediately executed in Morph's cloud environment.
3. Code Correction: If errors occur during code execution, it interprets the error messages and automatically corrects and re-executes the code. Corrections are made multiple times, autonomously fixing them until the execution is successful.

Code Generation and Correction

Morph AI’s code generation and correction feature supports implementation in SQL and Python editors. Understanding the data schema being used, it provides highly accurate implementation support. Additionally, it considers the type of database being used and the Python packages installed on Morph's cloud to generate code.

It offers features such as:

• Code Generation: Proposes new code that fits the user's requirements, considering the entire existing codebase.
• Code Correction: Proposes corrections only for specified parts of the code. This is effective when it is easier to specify parts of the code as prompts rather than expressing it as a full prompt.
• Error Correction: Proposes changes based on the location of the error and the error message if execution errors occur.

Data Q&A

Morph AI provides answers to users' questions about data in a conversational back-and-forth.

By specifying the tables to query and asking questions about the data, Morph AI automatically performs necessary preprocessing and aggregation to generate answers. This feature is useful for obtaining temporary aggregated results as it does not leave the process on the Canvas.

It can easily provide answers in cases such as:

1. Obtaining Aggregated Data Results: Answering business questions like, "tell me the ARR for the last six months," or, "tell me the channel with the most visitors to the LP." Just specify the necessary tables and ask the question.
2. Providing Answers Based on Visualization Results: Responding to requests like, "tell me about any abnormal cost increase rates recently," by visually capturing data and detecting changes. During the answer process, it creates graphs to visually assess the response, allowing simultaneous confirmation of the graphs that supported the answers.

Future Roadmap

With this update, Morph has undergone a fundamental reform and is moving towards public access. We plan to provide more features and content supporting more specific use cases.

To get the latest updates, follow us on social media or join our Discord community!