Vaibhav More Logo
VAIBHAVMORE
AI & NLP
8 min readNov 04, 2024

From Textbooks to Knowledge Graphs: Automated Semantic Entity Extraction with spaCy and Neo4j

How we parsed 200+ pages of academic syllabus material into interactive, queryable concept relationship graphs.

VM

Vaibhav More

Full-Stack Developer & AI/ML Enthusiast

#NLP#spaCy#KeyBERT#Neo4j#Python#Knowledge Graphs

Students and researchers frequently struggle with conceptual fragmentation: a machine learning textbook introduces 'Eigenvalues' in Chapter 2, references them in Chapter 7 under 'Principal Component Analysis', and assumes mastery in Chapter 11 for 'Spectral Clustering'. To bridge these knowledge gaps, we engineered the Knowledge Graph Platform: an automated NLP ingestion pipeline that turns raw documents into directed knowledge graphs.

The NLP Extraction Pipeline: Overcoming Tokenization & Chunking Limits

Ingesting academic PDFs requires more than basic string parsing. We utilized PyMuPDF for structural text block extraction, filtering headers, page numbers, and index footnotes.

Once cleaned, the document is segmented into semantically coherent paragraphs. We run a dual-pass linguistic analysis: spaCy's en_core_web_md handles Named Entity Recognition (NER) and syntactic dependency parsing, while KeyBERT extracts candidate technical keyphrases via BERT cosine similarity.

nlp/graph_extractor.pypython
import spacy
from keybert import KeyBERT

nlp = spacy.load("en_core_web_md")
kw_model = KeyBERT(model="all-MiniLM-L6-v2")

def extract_concepts_and_relations(text_chunk: str):
    doc = nlp(text_chunk)
    keywords = kw_model.extract_keywords(
        text_chunk, 
        keyphrase_ngram_range=(1, 3), 
        stop_words="english", 
        top_n=8
    )
    
    relations = []
    for token in doc:
        # Detect dependency verbs establishing directional relationships
        if token.pos_ == "VERB" and token.dep_ in ("ROOT", "relcl"):
            subjects = [w.text for w in token.lefts if w.dep_ in ("nsubj", "nsubjpass")]
            objects = [w.text for w in token.rights if w.dep_ in ("dobj", "pobj", "attr")]
            
            for s in subjects:
                for o in objects:
                    relations.append({
                        "source": s.strip(),
                        "relation": token.lemma_.upper(),
                        "target": o.strip()
                    })
    return keywords, relations

Graph Modeling in Neo4j with Cypher Primitives

Relational tables are notoriously inefficient for deep hierarchical concept traversals (requiring recursive CTE joins). In Neo4j, concepts become (:Concept) nodes, and relationships are typed edges: (:Concept)-[:REQUIRES]->(:Concept) or (:Concept)-[:EXTENDS]->(:Concept).

This enables lightning-fast shortest path queries: 'What are all mathematical prerequisites required before understanding Transformers?'

Cypher Query Efficiency

Using index-backed node lookup labels in Neo4j yields path traversal times under 8ms across 10,000+ relationships.

Key Architectural Takeaways

  • Combining spaCy dependency trees with KeyBERT embeddings produces high-precision technical relations.
  • Neo4j labeled property graphs provide natural visualization for academic prerequisites.
  • Containerizing the FastAPI extraction service and Neo4j instance ensures frictionless team deployment.

Conclusion & Next Steps

Knowledge graphs transform passive reading into active, exploratory learning. By combining lightweight BERT models with graph databases, self-directed learning paths can be mapped in real time.

More Engineering Deep Dives