apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-04-25 08:46:22 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Initial commit

Browse source
This commit is contained in:
Rohan Verma 2024-07-30 16:00:11 -07:00 • committed by GitHub
commit 55332d1ddb
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
168 changed files with 18456 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

161
apps/docs/documentation/configuration/docs.mdx Normal file
View file

@ -0,0 +1,161 @@
---
title: Documentation config
"og:title": Mintlify setup and configuration
description: Mintlify setup and configuration
---
## Introduction
The documentation site is built entirely using Mintlify. Mintlify docs are rendered from MDX files and configurations. Mintlify works with a Github integration to
constantly build an independent docs site based off your latest published repository.
Documentation can be edited in raw MDX, or directly through the Mintlify dashboard (which stays in sync with your local Github repo).
Mintlify is fast to setup and provides tonnes of features out-the-box without the need to pay for a subscription for the basic services.
You can run the **docs site only** by running the `pnpm run dev` command from a terminal within the `docs` directory.
## Docs structure
The docs root directory is structured as follows:
```
docs
├── _images
├── api
├── configuration
├── deployment
├── logos
├── *.mdx
├── mint.json
└── package.json
```
The docs directory is relatively simple and is made mostly of MDX files. Some notable files are:
| Item | Description |
| ------------ | ----------------------------------------------------------- |
| api | Automatically generated files (except for Introduction.mdx) |
| mint.json | Mintlify configuration object |
| package.json | Scripts needed in the monorepo/for generating the API files |
## Creating API endpoint documentation
We can leverage FastAPI's baked in OpenAPI support to automatically generate API documentation.
Using Mintlify's `@mintlify/scraping` package, we can scrape our FastAPI's OpenAPI schema to generate an interactive API documentation site.
<Tip>
For a more detailed description, read the [official
documentation](https://mintlify.com/docs/api-playground/openapi/setup#create-mdx-files-for-openapi-endpoints)
</Tip>
### Generating OpenAPI schema
#### Step 1: Generate MDX
Run the `generate-api` scripts within the `package.json` to generate the required MDX
files. There are two scripts here, one configured to use the production
`OpenAPI.json` and the second configured to use the development version.\
\
It is typically preferred to use the production version once any changes to
your API have been made and published.
#### Step 2: Update mint.json
<Steps>
<Step title="Add new MDX files to navigation">
Upon generating the relevant API information, `@mintlify/scraping` helpfully outputs the suggested navigation to the terminal:
```bash
navigation object suggestion:
[
{
"group": "users",
"pages": [
"api/users/get-user",
"api/users/get-all-users",
"api/users/search-users",
"api/users/create-user"
]
},
{
"group": "spells",
"pages": [
"api/spells/get-spell",
"api/spells/get-all-spells",
"api/spells/search-spells"
]
}
]
```
For sake of simplicity, copy this navigation object suggestion and add it your `mint.json` under the `navigation` object:
```json
"navigation": [
{
"group": "Getting Started",
"pages": ["documentation/introduction", "documentation/local-development"]
},
{
"group": "Configuration",
"pages": [
"documentation/configuration/turbo",
"documentation/configuration/fastapi",
"documentation/configuration/nextjs",
"documentation/configuration/docs"
]
},
{
"group": "Deployment",
"pages": [
"documentation/deployment/vercel",
"documentation/deployment/deployment"
]
},
{
"group": "API Reference",
"pages": ["api/introduction"]
},
{
"group": "Users",
"pages": [
"api/users/get-user",
"api/users/get-all-users",
"api/users/search-users",
"api/users/create-user"
]
},
{
"group": "Spells",
"pages": [
"api/spells/get-spell",
"api/spells/get-all-spells",
"api/spells/search-spells"
]
}
],
```
</Step>
<Step title="Add OpenAPI endpoint to `mint.json`">
Add the `openapi` key to your `mint.json`. This, preferably, can be linked to your published OpenAPI schema:
```json
{
...
"openapi":"https://next-fast-turbo-api.vercel.app/openapi.json"
...
}
```
This enables the generated MDX files to describe and interact with your API.
<Warning>
This has been hit and miss in the past for me. If doing this does not auto-populate your API reference with the documentation, the solution is to manually\
create an `openapi.json` file in the `docs` root. Manually copy and paste/save your `openapi.json` into this file.
</Warning>
</Step>
</Steps>

377
apps/docs/documentation/configuration/fastapi.mdx Normal file
View file

@ -0,0 +1,377 @@
---
title: FastAPI config
"og:title": "FastAPI configuration and setup"
description: FastAPI configuration and setup
---
## Introduction
The backend uses [FastAPI](https://fastapi.tiangolo.com/). FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.8+ based on standard Python type hints
You can run the **api only** by running the `pnpm run dev` command from a terminal within the `api` directory. Alternatively, you can directly run the `run.py` file.
## API structure
The API root directory is structured as follows:
```
api
├── src
│ ├── api
│ ├── crud
│ ├── schemas
│ ├── __init__.py
│ ├── config.py
│ └── main.py
├── tests
├── .env
├── .env.example
├── harry-potter-db-seed-spells.csv
├── harry-potter-db-seed-users.csv
├── package.json
├── poetry.lock
├── pyproject.toml
├── requirements.txt
├── run.py
└── vercel.json
```
### Src directory
The `src` directory is where all the application code sits. Below briefly explains each folder/file
| Item | Description |
| --------- | ----------------------------------------------- |
| api | The API endpoints for the application |
| crud | The CRUD operations used within the application |
| schemas | The schemas used within the application |
| config.py | Main application configuration |
| main.py | Application |
### Root directory
The root directory contains the typical files one would expect to see in a Python project. The below files are worth describing:
| Item | Description |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| package.json | Not typical in a Python programme; used due to the nature of the monorepo. Running `pnpm run dev` at the project root will execute the `dev` script in this `package.json` |
| vercel.json | The configuration for deploying the FastAPI aspect of the application to Vercel. |
| \*.csv | Simple seed data for the database, sourced from the [Harry Potter API](https://hp-api.onrender.com/) |
## Dependencies
### Python 3.9
Because the project is being deployed on Vercel, Python version `3.9` must be used as this is the latest supported version of Python. For more information, read the [official documentation](https://vercel.com/docs/functions/runtimes/python).
If deploying to somewhere other than Vercel, check which version of Python you may use and adjust the project according to your needs.
### Supabase
The project uses [Supabase](https://supabase.com/) as a database. Since Planetscale [removed their free Hobby tier](https://planetscale.com/blog/planetscale-forever),
Supabase has seemed like a good alternative to use. You do not have to use Supabase with the backend, but the project is written from the perspective of using it.
<Tip>
If you are using Supabase,
[`supabase-py-async`](https://pypi.org/project/supabase-py-async/) is already
included as a project dependency within the `pyproject.toml`. If you are not
using Supabase, this can be removed.
</Tip>
### Poetry
[Poetry](https://python-poetry.org/) is used to manage the virtual environment and project dependencies.
A `requirements.txt` has been generated to enable the installation of Python packages via the `pip install` command.
If you do not use Poetry, you can remove the `poetry.lock` and `pyproject.toml` files.
<Tip>You will need a `requirements.txt` when deploying. Vercel also accepts a `Pipenv` file if you use Pipenv, otherwise, you'll need the `requirements.txt` for the `api` to build correctly</Tip>
## Adding your own endpoints
Given the project structure, there are three areas that you must be aware of when adding your own endpoints with new models. The main areas are:
- Schemas
- CRUD
- API
### Example: Creation of Spells endpoints
#### Step 1: Create a schema
<Steps>
<Step title="Add a schema">
Add a new schema to the `schemas` directory.
```python src/schemas/spell.py
from typing import ClassVar, Sequence
from pydantic import BaseModel
class Spell(BaseModel):
id: str
name: str
description: str
table_name: ClassVar[str] = "spells"
class SpellCreate(BaseModel):
id: str
name: str
description: str
class SpellUpdate(BaseModel):
id: str
name: str
description: str
class SpellSearchResults(BaseModel):
results: Sequence[Spell]
```
<Info>The `table_name` value needs to be included in the base `Spell` class. This **must** be the same as the name of the table created in Supabase. It is used in the CRUD operations to identify the table to work with.</Info>
| Item | Description |
|--------------------|---------------------------------------------------------------------------------------------|
| `Spell` | The base class describing the columns that will be in the Supabase table |
| SpellCreate | The class for creating a new `Spell` |
| SpellUpdate | The class for updating an existing `Spell` |
| SpellSearchResults | Describes how data will be returned in the API response. It will be a `Sequence` of `Spell` |
</Step>
<Step title="Add to __init__.py">
To easily import the new `Spell` classes throughout our application, they need to be added to the `src/schemas/__init__.py` file.
```python src/schemas/__init__.py
from .user import User, UserCreate, UserSearchResults, UserUpdate
from .spell import Spell, SpellCreate, SpellSearchResults, SpellUpdate
```
This allows us to import from `src/schemas` like so:
```python
from src.schemas import Spell, SpellCreate, SpellSearchResults, SpellUpdate
```
</Step>
</Steps>
#### Step 2: Setup CRUD operations
Specific CRUD operations can be created for each endpoint. However, generic CRUD operations in `src/crud/base.py` can also be used without modification.
<Steps>
<Step title="Create CRUD file">
```python src/crud/crud_spell.py
from fastapi import HTTPException
from supabase_py_async import AsyncClient
from src.crud.base import CRUDBase
from src.schemas import Spell, SpellCreate, SpellUpdate
class CRUDSpell(CRUDBase[Spell, SpellCreate, SpellUpdate]):
async def get(self, db: AsyncClient, *, id: str) -> Spell | None:
try:
return await super().get(db, id=id)
except Exception as e:
raise HTTPException(
status_code=404,
detail=f"{e.code}: Spell not found. {e.details}",
)
async def get_all(self, db: AsyncClient) -> list[Spell]:
try:
return await super().get_all(db)
except Exception as e:
raise HTTPException(
status_code=404,
detail=f"An error occurred while fetching spells. {e}",
)
async def search_all(
self, db: AsyncClient, *, field: str, search_value: str, max_results: int
) -> list[Spell]:
try:
return await super().search_all(
db, field=field, search_value=search_value, max_results=max_results
)
except Exception as e:
raise HTTPException(
status_code=404,
detail=f"An error occurred while searching for spells. {e}",
)
spell = CRUDSpell(Spell)
```
The `CRUDSpell` class inherits from the `CRUDBase` class located in `src/crud/base.py`. The `Spell`, `SpellCreate` and `SpellUpdate` classes are passed to the `CRUDBase` class to specify the types of data that will be used in the CRUD operations.
The operations that will be used in the API endpoints are functions of the `CRUDSpell` class.
Finally, `spell` is an instance of `CRUDSpell`. We import this instance to use in the API endpoints like so: `spell.get_all(db)` or `spell.get(db, id=id)`.
<Note>
**This part of the project is intended to be read-only**, however, the `SpellCreate` and `SpellUpdate` classes are created in the schema regardless as they are required by the `CRUDBase` model's function arguments. `CRUDBase` can be refactored to have optional parameters, or a read-only version of the `CRUDBase` class can be created. This is beyond the scope of this project.
</Note>
</Step>
<Step title="Add to __init__.py">
Similar to setting up the schemas, the `CRUDSpell` class needs to be added to the `src/crud/__init__.py` file.
```python src/crud/__init__.py
from .crud_user import user
from .crud_spell import spell
```
</Step>
</Steps>
#### Step 3: Create the API endpoints
<Steps>
<Step title="Create the endpoints">
Create the endpoints in the `src/api/api_v1/endpoints/spells.py` file.
```python src/api/api_v1/endpoints/spells.py
from typing import Literal, Optional, Union
from fastapi import APIRouter, HTTPException
from src.api.deps import SessionDep
from src.crud import spell
from src.schemas import Spell, SpellSearchResults
router = APIRouter()
@router.get("/get/", status_code=200, response_model=Spell)
async def get_spell(session: SessionDep, spell_id: str) -> Spell:
"""Returns a spell from a spell_id.
**Returns:**
- spell: spell object.
"""
return await spell.get(session, id=spell_id)
@router.get("/get-all/", status_code=200, response_model=list[Spell])
async def get_all_spells(session: SessionDep) -> list[Spell]:
"""Returns a list of all spells.
**Returns:**
- list[spell]: List of all spells.
"""
return await spell.get_all(session)
@router.get("/search/", status_code=200, response_model=SpellSearchResults)
async def search_spells(
session: SessionDep,
search_on: Literal["spells", "description"] = "spell",
keyword: Optional[Union[str, int]] = None,
max_results: Optional[int] = 10,
) -> SpellSearchResults:
"""
Search for spells based on a keyword and return the top `max_results` spells.
**Args:**
- keyword (str, optional): The keyword to search for. Defaults to None.
- max_results (int, optional): The maximum number of search results to return. Defaults to 10.
- search_on (str, optional): The field to perform the search on. Defaults to "email".
**Returns:**
- SpellSearchResults: Object containing a list of the top `max_results` spells that match the keyword.
"""
if not keyword:
results = await spell.get_all(session)
return SpellSearchResults(results=results)
results = await spell.search_all(
session, field=search_on, search_value=keyword, max_results=max_results
)
if not results:
raise HTTPException(
status_code=404, detail="No spells found matching the search criteria"
)
return SpellSearchResults(results=results)
```
Here we are calling `spell` object which we setup in step 2.
The `SessionDep` (located in `src/api/deps.py`) dependency is used to access the database.
The `response_model` parameter is used to specify the type of data that will be returned from the endpoint. This is used to generate TypeScript types for the frontend.
</Step>
<Step title="Add the endpoint to the router">
To include the new endpoints in the API, it must be added to the API router, located in `src/api/api_v1/api.py`.
```python src/api/api_v1/api.py
from fastapi import APIRouter
from src.api.api_v1.endpoints import users, spells
api_router = APIRouter()
api_router.include_router(users.router, prefix="/users", tags=["users"], responses={404: {"description": "Not found"}})
api_router.include_router(spells.router, prefix="/spells", tags=["spells"], responses={404: {"description": "Not found"}})
```
</Step>
</Steps>
#### Step 4: Create the table in Supabase
<Steps>
<Step title="Create the table">
Create a new table in your Supabase dashboard. It is **imperative** that the table name matches the `table_name` value in the `Spell` class in `src/schemas/spell.py`.
Columns should be added to match the `Spell` class. For example, we have the following `Spell` class:
```python
class Spell(BaseModel):
id: str
name: str
description: str
table_name: ClassVar[str] = "spells"
```
In this example, the table should be named `spells` (table names are case-sensitive) with the columns `id`, `name`, and `description`. \
\
For data types, `id` can be automatically assigned - in this example, it is set to a `uuid`. The `name` and `description` fields are strings, therefore their column's type is set to `text`. The `id` column should be the primary key.
<Tip>You do not need to add the `table_name` column as this is used in the FastAPI code to identify the table to work with.</Tip>
</Step>
<Step title="Seed the table with data">
The data can be seeded using the Supabase dashboard by uploading the `harry-potter-db-seed-spells.csv` file. For a more detailed explanation, please see the [official documentation](https://supabase.com/docs/guides/database/import-data#option-1-csv-import-via-supabase-dashboard)
</Step>
<Step title="Configure table security">
The table security should be configured to allow the FastAPI application to access the data. By default, Supabase will have [RLS (Row Level Security)](https://supabase.com/docs/guides/auth/row-level-security) enabled.\
\
If left un-configured, the database will return an empty array. As this is a simple read-only project, I am turning `RLS` off. However, for true CRUD operations, it should be configured to use authentication which is beyond the scope of this project.
\
To disable `RLS`, in the table settings, select `configure RLS` and then `disable RLS`.\
</Step>
<Step title="Add table connection to .env">
Once the table has been created, you must ensure that the `DB_URL` and `DB_API_KEY` parameters are populated in your `.env` file located in the root of the `API` directory.\
\
These values come from the Supabase dashboard by going to `settings` and then `API`. Copy the Project URL (`DB_URL`) and the Project API Key (`DB_API_KEY`).
<Info>
`DB_USERNAME` and `DB_PASSWORD` should also included in the `.env` as they are configured in the `src/config.py`. If you do not include these you will receive an error from Pydantic.\
\
Their inclusion is a pre-cursor to authentication, but they are not actually used in this project scaffold.
</Info>
</Step>
</Steps>
#### Step 5: Test your new endpoints
You can now test your new endpoints using the FastAPI Swagger UI or by making requests to the API.

176
apps/docs/documentation/configuration/nextjs.mdx Normal file
View file

@ -0,0 +1,176 @@
---
title: Next.js config
"og:title": "Next.js config and setup"
"description": "Next.js config and setup"
---
## Introduction
The frontend uses the latest version of Next.js and TypeScript; it is configured using the latest [app router](https://nextjs.org/docs/app).
You can run the **frontend only** by running the `pnpm run dev` command from a terminal within the `web` directory. Alternatively, you can run the entire application by running `pnpm run dev` from the **root directory**.
## App structure
The app root directory is structured as follows:
```
web
├── app
├── components
├── lib
├── public
├── .env
├── .env.example
├── .eslintrc.js
├── components.json
├── next.config.js
├── package.json
├── postcss.config.js
├── README.md
├── tailwind.config.js
├── tsconfig.json
└── yarn.lock
```
#### App
| Item | Description |
| ----------- | ---------------------------------------------------------------------------------------------------- |
| globals.css | Tailwind global config. Uses a generated theme from [ShadCN UI themes](https://ui.shadcn.com/themes) |
| layout.tsx | App layout |
| page.tsx | Landing page |
#### Components
| Item | Description |
| ------- | ----------------------------------------------------------------------------------------------------------------------------- |
| layouts | The dashboard layout is stored which wraps the application `{children}` in `root/app/layout.tsx` |
| theme | The `ThemeProvider` component(s) that wrap the application. |
| ui | Where `ShadCN` components are stored upon being generated. Configured by the `components.json` located in the `frontend root` |
| \*.tsx | Components used within the application |
#### Lib
| Item | Description |
| ----------- | ---------------------------------------------------------------------------------------------------------------- |
| api | The output generated by the `openapi` npm package. Configured in the `package.json` task named `generate-client` |
| config | Stores site config. Currently contains the setup for sidebar navigation |
| twConfig.ts | Tailwind colour config exported to an accessible Object. Used for assigning theme-aware colours in charts |
| utils.ts | ShadCN utils |
## TypeScript from FastAPI
As FastAPI is based on the OpenAPI specification, you get automatic compatibility with many tools, including the automatic API docs (provided by Swagger UI).
One particular advantage that is not necessarily obvious is that you can generate clients (sometimes called SDKs ) for your API, for many different programming languages.
<Note>For a more detailed explanation, see the [official FastAPI documentation](https://fastapi.tiangolo.com/advanced/generate-clients/#generate-a-typescript-client-with-the-preprocessed-openapi)</Note>
## Why generate TypeScript?
Generating a TypeScript client for your FastAPI backend is a great way to ensure that your frontend and backend are always in sync. It is also a way to provide type hinting while writing your frontend code without needing a permanent reference to the API, or re-creating the schema using something like Zod.
<Frame caption="Type hinting from the generated API client"><img src="/_images/type-hinting.png" /></Frame>
### How to generate TypeScript
<Steps>
<Step title="Configure and run generate-client task">
There are two generate-client tasks configured in the `package.json`:
```JSON
"scripts": {
...
"generate-client": "openapi --input https://next-fast-turbo.vercel.app/openapi.json --output ./lib/api/client --client axios --useOptions --useUnionTypes",
"generate-client:dev": "openapi --input http://127.0.0.0:8000/openapi.json --output ./lib/api/client --client axios --useOptions --useUnionTypes"
...
},
```
The `generate-client` task is set to run off the production OpenAPI schema.\
\
The `generate-client:dev` task is set to use the localhost OpenAPI schema. This is useful for development, as it will use the latest schema from the backend.
Ensure your production API URL is configured, or that your local API URL is correct, and then run the relevant task.
</Step>
<Step title="Update your root layout">
A key file to be aware of is the `OpenAPI.ts` which is generated in the `lib/api/client/core` directory. This file has the main configuration for the API connection as below:
```tsx lib/api/client/core/OpenAPI.ts
export const OpenAPI: OpenAPIConfig = {
BASE: "http://127.0.0.0:8000",
VERSION: "0.1.0",
WITH_CREDENTIALS: false,
CREDENTIALS: "include",
TOKEN: undefined,
USERNAME: undefined,
PASSWORD: undefined,
HEADERS: undefined,
ENCODE_PATH: undefined,
};
```
The problem with this is that we would want to differentiate between which API is being used. When the TypeScript is generated, the given OpenAPI URL is used as the base URL for the API.
This is not ideal for production, as you would want to use the production API URL, and vice versa in development.
You can override this by adding the following to your root `layout.tsx` file:
```tsx layout.tsx
import { OpenAPI } from "@/lib/api/client";
if (process.env.NODE_ENV === "production") {
OpenAPI.BASE = "https://next-fast-turbo.vercel.app"
}
```
</Step>
</Steps>
## Using the generated API
Once you have generated your TypeScript interface to your API, you can use it in your frontend code.
In its simplest form, you can import the generated API and use it as below:
```tsx
import { UsersService } from "@/lib/api/client";
const response = await UsersService.usersSearchUsers({
keyword: "keyword",
searchOn: "searchOn",
maxResults: "maxResults",
});
```
You can see this code being used in the `components/search-users.tsx` file, as part of the form's onSubmit function:
```tsx
const onSubmit = async (data: z.infer<typeof FormSchema>) => {
try {
const maxResults = data.searchResults ? parseInt(data.searchResults) : 10;
const response = await UsersService.usersSearchUsers({
keyword: data.keyword,
searchOn: data.searchOn,
maxResults: maxResults,
});
setSearchResults(response);
setError(null);
} catch (error) {
setSearchResults({ results: [] });
setError(error);
}
};
```
This code also imports the `UserSearchResults` type which is used to set the state of the search results:
```tsx
import { UsersService, UserSearchResults } from "@/lib/api/client";
const [searchResults, setSearchResults] = React.useState<UserSearchResults>({
results: [],
});
```
## Important considerations
- If you make changes to your API, you will need to re-generate the TypeScript interface.
- Re-generating this interface will overwrite the existing files. If you want to modify anything (e.g. `api/client/core/OpenAPI.ts`) and have it persist, do this outside of the generated files. You can see this in action in the `root/app/layout.tsx` file.

53
apps/docs/documentation/configuration/turbo.mdx Normal file
View file

@ -0,0 +1,53 @@
---
title: Turbo config
"og:title": "Configuring Turbo for FastAPI, Next.js and Turborepo"
description: Configuring Turbo for FastAPI, Next.js and Turborepo.
---
## Introduction
This project was first created using the `create-turbo` npm package:
<CodeGroup>
```bash npm
npx create-turbo@latest
```
```bash yarn
yarn dlx create-turbo@latest
```
```bash pnpm
pnpm dlx create-turbo@latest
```
</CodeGroup>
<Note>It is recommended that you read through the [official documentation](https://turbo.build/repo/docs/getting-started/create-new) to understand the breakdown of the monorepo.</Note>
## Changes to the official Turbo starter
The official Turbo starter is a great starting point for a monorepo, but it has some configurations I did not need. As I will not be sharing code between the backend and frontend (nor the docs), I have **removed** the `packages/ui` directory.
If you want to **add a new package** to the project, please reference the [official documentation](https://turbo.build/repo/docs/handbook/sharing-code/internal-packages).
For quick reference:
1. Create a new folder in `packages/<folder>`
2. Add a `package.json`, with `name` and `types` pointing at `src/index.ts[x]`
3. Add `src/index.ts[x]`, with at least one named export
4. Install your packages from `root`
## Turbo.json
The currently configured `turbo.json` does not have many changes from the official starter. The only change is the addition of the `globalEnv` key.
```json
{
"globalEnv": [
"NODE_ENV"
],
}
```
Without adding these environment variables to the `globalEnv`, you will receive an eslint error when referencing variables in your `.env`, however, your code will still work.
<Warning>
There are different ways that people solve this. While it's possible to use\
`"globalDependencies": ["**/.env"]`, that did not work in this project.\
</Warning>