In this project we will evaluate the quality of all the articles submitted by the content writers. All the articles are fetched based on a time window (last 3 days, last one week etc). Llamaindex workflow will be utilized to check each article for its structure, any missing elements are there in the article and give an overall feedback. We use a pydantic schema to predefine the structure and llm will be using the sturcture predict to evaluate. A final report is created.
First approach, each is a seperate task
+----------------------------------+
| User Time Window |
| (e.g., "Last 24 Hours") |
+----------------+-----------------+
|
v
+----------------+-----------------+
| CMS System |
| (Fetch Articles in Time Window) |
+----------------+-----------------+
|
v
+-------------------------------------------------------------------+
| LLAMAINDEX WORKFLOW |
| |
| +-----------------------+ +--------------------+ |
| | Start Event | | Router / Dispatch | |
| | (Incoming CMS Data) +------------->| (Parallel Execution) | |
| +-----------------------+ +---------+----------+ |
| | |
| +-------------------+----------------------+ |
| | | | |
| v v v |
| +--------------+ +--------------+ +-------------------+ |
| | Task 1: | | Task 2: | | Task 3: | |
| | Structure | | Wellformed | | Content Quality | |
| | Analysis | | Validation | | Assessor | |
| +------+-------+ +------+-------+ +---------+---------+ |
| | | | |
| +-------------------+----------------------+ |
| | (Emit Partial Results) |
| v |
| +--------------------+ |
| | Gather / Step | |
| | (Collects 3 Tasks) | |
| +----------+---------+ |
| | |
| v |
| +--------------------+ |
| | Stop / Report Step| |
| | (Synthesize Data) | |
| +----------+---------+ |
+-----------------------------|-------------------------------------+
|
v
+--------------+-------------------+
| Final Quality Report |
| (Structure, Format & Insights) |
+----------------------------------+
Second approach is all the tasks are clubbed to one single workflow step task.
+----------------------------------+
| User Time Window |
| (e.g., "Last 24 Hours") |
+----------------+-----------------+
|
v
+----------------+-----------------+
| CMS System |
| (Fetch Articles in Time Window) |
+----------------+-----------------+
|
v
+-------------------------------------------------------------------+
| SINGLE-STEP STRUCTURED PREDICT |
| |
| +-----------------------+ +--------------------+ |
| | Raw Article Data +------------->| LLM Context | |
| +-----------------------+ | (Single Payload) | |
| +---------+----------+ |
| | |
| v |
| +--------------------+ |
| |llm.structured_predict |
| +---------+----------+ |
| | |
| v |
| +-----------------+------------------+
| | Unified Pydantic Schema Validation|
| | [Structure + Format + Quality] |
| +-----------------+------------------+
+----------------------------------------------------|--------------+
|
v
+-------------------------------------+------------+
| Final Article Report |
| (Instantly validated JSON / Typed Python Object) |
+--------------------------------------------------+
Let's begin building the project. Code is organized as below.
├── article_schema.py --> Pydantic Schema for the structure
├── audit_workflow.py
├── create_report.py
├── fetch_articles.py --> Fetches the articles from a CMS
from pydantic import BaseModel, Field
from typing import List, Optional
#consider an article with an id, title, description fields
class ArticleStructure(BaseModel):
collectionId: str = Field(description="The unique identifier of the article.")
title: str = Field(short_summary="Check if title is engaging and properly cased.")
well_formed: bool = Field(description="Overall structural integrity of the article.")
missing_elements: Optional[List[str]] = Field(description="List of structural elements that are missing, if any.")
feedback: str = Field(description="Actionable feedback for the author.")
Test the schema, by creating some dummy data point.
dummy_article = ArticleStructure(collectionId="1",title="some_thing", well_formed="yes", missing_elements=[], feedback="something")
print(dummy_article)
Now, let's get the ball rolling and fetch some articles from a CMS system. If you don't have one, you may use some fastapi to implement a dummy article repository.
import requests
from datetime import datetime, timedelta, timezone
def fetch_recent_articles(pocketbase_url: str):
# Calculate time threshold (e.g., 3 hours ago)
time_threshold = datetime.now(timezone.utc) - timedelta(days=50)
time_string = time_threshold.isoformat().replace("+00:00", "Z")
# Query PocketBase API, giving the full path where articles reside
filter_query = f"updated >= '{time_string}'"
url = f"{pocketbase_url}/api/collections/nature_posts/records"
params = {"filter": filter_query}
response = requests.get(url, params=params)
response.raise_for_status()
return response.json().get("items", [])
Now, let's write a test audit program, we will use a locally hosted llm using openAI like.
import json
from llama_index.llms.openai_like import OpenAILike
import article_schema as asc
from llama_index.core.prompts import PromptTemplate
#creating a dummy article to test it
custom_llm = OpenAILike(model="gemma", api_base="http://localhost:8080/v1", api_key="not_needed", is_chat_model=True)
prompt = f"""
Analyze the following article for well-formed structure, tone, and readability.
CollectionId: {'something'}
Title: {'Rose, a beautiful plant'}
Content: {'Roses are very beautiful plants and grow in different colors. They are great addition to any garden! Roses are popular woody shrubs that belong to the genus Rosa.They grow colorful flowers in shades of red, pink, yellow, and white.Their stems have sharp prickles, commonly called thorns, to protect against animals.Most rose varieties produce a sweet and pleasant fragrance.'}
"""
# Use structured output to force the Pydantic schema evaluation
response = custom_llm.structured_predict(
output_cls=asc.ArticleStructure,
prompt=PromptTemplate(prompt)
)
print(response)
If everything goes well, you should be able to see something like this.
collectionId='CollectionId' title='Rose, a beautiful plant' well_formed=True missing_elements=[] feedback='The article is well-formed with a clear structure. It follows a simple, direct narrative. The title is engaging and appropriately formatted. The content is concise and informative, providing a basic overview of roses. There are no missing elements.'
Expand on the concept and implement a simple workflow using llamaindex to complete the quality check of the contents of the articles.