Large-Scale Academic Intelligence Database for Researcher Ranking

Affiliation: Chinese Academy of Sciences (CAS) · Intern Researcher
Role: Database Architect & Data Pipeline Engineer
Scale: 160M+ academic records · OpenAlex dataset

Overview

This project builds the data infrastructure underpinning a scientific impact evaluation and tech-talent ranking system, developed in collaboration with the BenchCouncil research group at CAS. The system processes the full OpenAlex academic graph to construct structured practitioner profiles, enabling downstream ranking and classification tasks based on research contribution and topic expertise.

The final system was deployed to produce the BenchCouncil AI Future Stars Ranking (Nov 2025), covering 1,867 early-career researchers across the global AI community.

Database Design

The core challenge was modeling the full academic graph — works, authors, institutions, topics, citations, and affiliations — in a way that supports efficient large-scale relational queries under strict memory constraints. The schema was designed around OpenAlex's data model with careful normalization to avoid redundancy at scale.

Key design decisions:

• Mapping tables (WORK_AUTHOR, WORK_INSTITUTION, etc.) decompose all many-to-many relationships, keeping core entity tables narrow and index-friendly
• The topic taxonomy (DOMAIN → FIELD → subfield) is stored as explicit relationship tables rather than nested JSON, enabling efficient hierarchical queries without loading full subtrees into memory
• WORK_REFERENCE and WORK_RELATED model the citation graph as self-referential edge tables, supporting graph-based influence analysis

Engineering Challenges

Challenge 1 — Ingesting 160M Records under Memory Constraints

The OpenAlex snapshot is distributed across thousands of compressed JSONL files totalling hundreds of gigabytes. Sequential download and ingestion was infeasible given time and memory constraints on the available server.

Solution: Designed a concurrent multi-process pipeline that parallelized both the download and the PostgreSQL bulk-insert stages. Each worker process handled an independent file shard — decompressing, parsing, and batch-inserting records via COPY rather than row-level INSERT. Custom composite indexes were designed for the most frequent join patterns (e.g., author–work–institution chains), sized to fit within the server's physical memory budget rather than defaulting to full-table indexes.

Challenge 2 — llm2vec Deployment on V100 GPUs

To generate semantic vector representations for academic entities, we deployed llm2vec for embedding author research profiles. However, llm2vec's default multi-GPU inference mode is incompatible with V100 architecture on the Zhijiang cluster (due to unsupported mixed-precision operations).

Solution: Reconfigured the inference pipeline to run in single-GPU mode with explicit dtype and device mapping overrides, and compensated for the reduced throughput by writing concurrent multi-processing scripts that dispatched independent embedding jobs across multiple single-GPU processes in parallel. This recovered the throughput loss while remaining within V100 hardware constraints.

Deployment

The infrastructure was used to produce the BenchCouncil AI Future Stars Ranking (November 2025), a public index of 1,867 early-career AI researchers evaluated using the Evaluatology-based methodology developed by the BenchCouncil group. The ranking covers institutions including Tsinghua University, NTU, Google, Peking University, and CAS, among others.

Note: My contribution to this project was the database architecture and data pipeline engineering. The ranking methodology and downstream evaluation algorithms were designed by the BenchCouncil research team.