Spaces:
Runtime error
Runtime error
Shivam Kumar
commited on
Commit
β’
344e641
1
Parent(s):
82802ba
π Typo correction and Grammatical correction in the `developing` documentation page (#360)
Browse files- docs/developing.md +82 -82
docs/developing.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1 |
# Developing
|
2 |
|
3 |
-
This page of the docs outlines how to get **Websurfx** up and running in a development environment, and outlines the common
|
4 |
|
5 |
<details>
|
6 |
<summary><b>Table of Contents</b></summary>
|
@@ -30,50 +30,50 @@ This page of the docs outlines how to get **Websurfx** up and running in a devel
|
|
30 |
|
31 |
## Setting up the Development Environment
|
32 |
|
33 |
-
By default, we provide four different ways to work on the project.
|
34 |
|
35 |
- [Local Development](#local-development-)
|
36 |
- [Gitpod](#gitpod-)
|
37 |
- [NixOS Dev Shell using Nix Flake](#nixos-dev-shell-using-nix-flake-)
|
38 |
- [Local Development with Docker Compose](#local-development-with-docker-compose-)
|
39 |
|
40 |
-
The
|
41 |
|
42 |
### Local Development
|
43 |
|
44 |
-
This section covers how to
|
45 |
|
46 |
#### Prerequisites
|
47 |
|
48 |
Before you start working on the project. You will need the following packages installed on your system:
|
49 |
|
50 |
-
-
|
51 |
-
-
|
52 |
-
-
|
53 |
-
-
|
54 |
|
55 |
```shell
|
56 |
$ npm i -g stylelint
|
57 |
```
|
58 |
|
59 |
> **Note**
|
60 |
-
> In the above command the dollar sign(**$**) refers to running the command in
|
61 |
|
62 |
-
- `Cargo-watch` installed on your system which will allow to auto
|
63 |
|
64 |
```shell
|
65 |
cargo install cargo-watch
|
66 |
```
|
67 |
|
68 |
-
- `Git` installed on your system. The installation
|
69 |
-
- Finally,
|
70 |
|
71 |
> **Note**
|
72 |
-
> For **rolling release
|
73 |
>
|
74 |
> **For Example:**
|
75 |
>
|
76 |
-
> On `arch linux` the following packages
|
77 |
>
|
78 |
> - `Cargo`: https://wiki.archlinux.org/title/rust
|
79 |
> - `Npm`: https://wiki.archlinux.org/title/Node.js
|
@@ -81,24 +81,24 @@ cargo install cargo-watch
|
|
81 |
> - `Git`: https://wiki.archlinux.org/title/git
|
82 |
> - `Docker`: https://wiki.archlinux.org/title/docker
|
83 |
>
|
84 |
-
> But we do not recommend this method for **stable release
|
85 |
|
86 |
#### Setting up Pre-commit Checks
|
87 |
|
88 |
-
Before you
|
89 |
|
90 |
```shell
|
91 |
git clone https://github.com/<your_github_username>/websurfx.git
|
92 |
cd websurfx
|
93 |
```
|
94 |
|
95 |
-
Once you have
|
96 |
|
97 |
```shell
|
98 |
cargo test
|
99 |
```
|
100 |
|
101 |
-
By
|
102 |
|
103 |
#### Running the Project
|
104 |
|
@@ -108,7 +108,7 @@ If you have followed the above section then you should have a cloned repository
|
|
108 |
cargo watch -q -x "run" -w "."
|
109 |
```
|
110 |
|
111 |
-
This will compile the app by default with the **In-Memory caching** feature. To compile, run and test the app with other features
|
112 |
|
113 |
##### Hybrid Cache
|
114 |
|
@@ -140,13 +140,13 @@ cargo watch -q -x "run --no-default-features --features redis-cache" -w .
|
|
140 |
> redis-server --port 8082 &
|
141 |
> ```
|
142 |
|
143 |
-
Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to
|
144 |
|
145 |
> For more info on all the project commands. See: [**Project Commands**](#project-commands-)
|
146 |
|
147 |
### Gitpod
|
148 |
|
149 |
-
This section covers
|
150 |
|
151 |
> **Note**
|
152 |
> By default the project only supports the Vscode **IDE/Editor** for Gitpod.
|
@@ -167,7 +167,7 @@ https://gitpod.io/#https://github.com/<your_github_username>/websurfx
|
|
167 |
|
168 |
The project by default provides a set of pre-installed plugins for gitpod which is done to improve productivity and efficiency while working on the project. Also to make working on the project more fun and engaging which can be customized from within the `Gitpod` instance.
|
169 |
|
170 |
-
The list of all the pre-installed plugins are listed
|
171 |
|
172 |
**Productivity**
|
173 |
|
@@ -191,7 +191,7 @@ The list of all the pre-installed plugins are listed down below:
|
|
191 |
- [Test Explorer UI](https://open-vsx.org/extension/hbenl/vscode-test-explorer): Provides a way to run any test from a convenient sidebar.
|
192 |
- [GitLens](https://open-vsx.org/extension/eamodio/gitlens): Provides a better and more efficient way to manage common git workflows.
|
193 |
|
194 |
-
> Optionally, if you prefer more keyboard
|
195 |
>
|
196 |
> - [VSCode Neovim](https://open-vsx.org/extension/asvetliakov/vscode-neovim): Provides complete vim emulation for vscode.
|
197 |
|
@@ -204,16 +204,16 @@ The list of all the pre-installed plugins are listed down below:
|
|
204 |
|
205 |
### NixOS Dev Shell using Nix Flake
|
206 |
|
207 |
-
This section covers
|
208 |
|
209 |
#### Pre Setup Requirements
|
210 |
|
211 |
Before you start working on the project. You will need the following packages installed on your system:
|
212 |
|
213 |
-
- `Git` installed on your system. The installation
|
214 |
-
- Finally,
|
215 |
|
216 |
-
> Optionally, On `NixOS` the above
|
217 |
>
|
218 |
> - `Git`: https://search.nixos.org/packages?channel=23.05&show=git&from=0&size=50&sort=relevance&type=packages&query=git
|
219 |
> - `Docker`: https://search.nixos.org/packages?channel=23.05&show=docker&from=0&size=50&sort=relevance&type=packages&query=docker
|
@@ -233,19 +233,19 @@ Then run the following command to setup the `NixOS dev-shell`:
|
|
233 |
nix develop
|
234 |
```
|
235 |
|
236 |
-
Once you have
|
237 |
|
238 |
```shell
|
239 |
cargo test
|
240 |
```
|
241 |
|
242 |
-
By
|
243 |
|
244 |
#### Post Setup Requirements
|
245 |
|
246 |
-
After you have done setting up pre-commit checks, then you may need to
|
247 |
|
248 |
-
- `Cargo-watch` installed on your system which will allow to auto
|
249 |
|
250 |
```shell
|
251 |
cargo install cargo-watch
|
@@ -259,7 +259,7 @@ If you have followed the above section then you should now be inside a `dev-shel
|
|
259 |
cargo watch -q -x "run" -w "."
|
260 |
```
|
261 |
|
262 |
-
This will compile the app by default with the **In-Memory caching** feature. To compile, run and test the app with other features
|
263 |
|
264 |
##### Hybrid Cache
|
265 |
|
@@ -285,49 +285,49 @@ To build the search engine with the `Redis caching` feature. Run the following c
|
|
285 |
cargo watch -q -x "run --no-default-features --features redis-cache" -w .
|
286 |
```
|
287 |
|
288 |
-
> Optionally, If you have build and run the app with the `Redis cache`or `Hybrid cache` feature (as mentioned above) then you will need to start the redis server alongside the app which can be done
|
289 |
>
|
290 |
> ```shell
|
291 |
> redis-server --port 8082 &
|
292 |
> ```
|
293 |
|
294 |
-
Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to
|
295 |
|
296 |
### Local Development with Docker Compose
|
297 |
|
298 |
-
This section covers how to
|
299 |
|
300 |
#### Prerequisites
|
301 |
|
302 |
Before you start working on the project. You will need the following packages installed on your system:
|
303 |
|
304 |
-
-
|
305 |
-
-
|
306 |
-
-
|
307 |
|
308 |
```shell
|
309 |
$ npm i -g stylelint
|
310 |
```
|
311 |
|
312 |
> **Note**
|
313 |
-
> In the above command the dollar sign(**$**) refers to running the command in
|
314 |
|
315 |
-
- `Git` installed on your system. The installation
|
316 |
-
- Finally,
|
317 |
|
318 |
> **Note**
|
319 |
-
> For **rolling release
|
320 |
>
|
321 |
> **For Example:**
|
322 |
>
|
323 |
-
> On `arch linux` the following packages
|
324 |
>
|
325 |
> - `Cargo`: https://wiki.archlinux.org/title/rust
|
326 |
> - `Npm`: https://wiki.archlinux.org/title/Node.js
|
327 |
> - `Git`: https://wiki.archlinux.org/title/git
|
328 |
> - `Docker`: https://wiki.archlinux.org/title/docker
|
329 |
>
|
330 |
-
> But we do not recommend this method for **stable release
|
331 |
|
332 |
#### Setting up Pre-commit Checks
|
333 |
|
@@ -338,13 +338,13 @@ git clone https://github.com/<your_github_username>/websurfx.git
|
|
338 |
cd websurfx
|
339 |
```
|
340 |
|
341 |
-
Once you have
|
342 |
|
343 |
```shell
|
344 |
cargo test
|
345 |
```
|
346 |
|
347 |
-
By
|
348 |
|
349 |
#### Running the Project
|
350 |
|
@@ -355,9 +355,9 @@ $ docker compose -f dev.docker-compose.yml up
|
|
355 |
```
|
356 |
|
357 |
> **Note**
|
358 |
-
> In the above command the dollar sign(**$**) refers to running the command in
|
359 |
|
360 |
-
Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to
|
361 |
|
362 |
### Project Commands
|
363 |
|
@@ -383,12 +383,12 @@ Once you have finished running the above command, Websurfx should now be served
|
|
383 |
|
384 |
### Environment Variables
|
385 |
|
386 |
-
All environment variables are optional. Currently there are not many environment variables used, as most of the user preferences are stored under the `websurfx` folder (located under the codebase (`websurfx` directory)) in the `config.lua` file.
|
387 |
|
388 |
-
The list of all the available environment variables are listed
|
389 |
|
390 |
-
- `PKG_ENV`: Sets the logging level for the app to **Trace** which can be useful for better debugging of the app.
|
391 |
-
- `RUST_BACKTRACE`: Rust
|
392 |
|
393 |
## Git Strategy
|
394 |
|
@@ -396,9 +396,9 @@ The list of all the available environment variables are listed down below:
|
|
396 |
|
397 |
Like most Git repos, we are following the [Github Flow](https://guides.github.com/introduction/flow) standard.
|
398 |
|
399 |
-
1. Create a branch (or fork if you don'
|
400 |
2. Code some awesome stuff π§βπ»
|
401 |
-
3. Add, commit and push your changes to your branch/ fork
|
402 |
4. Head over to GitHub and create a Pull Request
|
403 |
5. Fill in the required sections in the template, and hit submit
|
404 |
6. Follow up with any reviews on your code
|
@@ -411,9 +411,9 @@ For example, `FEATURE/420_Awesome-feature` or `FIX/690_login-server-error`
|
|
411 |
|
412 |
### Commit Emojis
|
413 |
|
414 |
-
Using a single emoji at the start of each commit message, issue title, and pull request title, to indicate the type task, makes the commit ledger, issue and pull request easier to understand,
|
415 |
|
416 |
-
- π¨ `:art:` - Improve structure
|
417 |
- β‘οΈ `:zap:` - Improve performance.
|
418 |
- π₯ `:fire:` - Remove code or files.
|
419 |
- π `:bug:` - Fix a bug.
|
@@ -426,15 +426,15 @@ Using a single emoji at the start of each commit message, issue title, and pull
|
|
426 |
- β
`:white_check_mark:` - Add, update, or pass tests.
|
427 |
- ποΈ `:lock:` - Fix security issues.
|
428 |
- π `:bookmark:` - Make a Release or Version tag.
|
429 |
-
- π¨ `:rotating_light:` - Fix compiler
|
430 |
- π§ `:construction:` - Work in progress.
|
431 |
- β¬οΈ `:arrow_up:` - Upgrade dependencies.
|
432 |
-
- π· `:construction_worker:` - Add or update CI build system.
|
433 |
- β»οΈ `:recycle:` - Refactor code.
|
434 |
- π©Ή `:adhesive_bandage:` - Simple fix for a non-critical issue.
|
435 |
- π§ `:wrench:` - Add or update configuration files.
|
436 |
- π± `:bento:` - Add or update assets.
|
437 |
-
- ποΈ `:card_file_box:` - Perform database schema
|
438 |
- βοΈ `:pencil2:` - Fix typos.
|
439 |
- π `:globe_with_meridians:` - Internationalization and translations.
|
440 |
|
@@ -446,26 +446,26 @@ Once you've made your changes, and pushed them to your fork or branch, you're re
|
|
446 |
|
447 |
For a pull request to be merged, it must:
|
448 |
|
449 |
-
- The build, lint and tests (run by GH actions) must pass
|
450 |
- There must not be any merge conflicts
|
451 |
|
452 |
When you submit your pull request, include the required info, by filling out the pull request template. Including:
|
453 |
|
454 |
- A brief description of your changes.
|
455 |
- The issue or ticket number (if applicable).
|
456 |
-
- For UI
|
457 |
-
- If any dependencies were added, explain why it was needed, state the cost. associated, and confirm it does not introduce any security, privacy or speed issues
|
458 |
-
- Optionally, provide a checklist of all the changes that
|
459 |
|
460 |
> **Important**
|
461 |
-
> Make sure to fill all the required/mandatory sections of the pull request as
|
462 |
|
463 |
> **Note**
|
464 |
> The pull request template contains comments in the following form `<!-- -->` which are used to provide a guide on what should be provided under each heading of the template. These comments are never rendered when the pull request is either created or updated and hence anything provided in such comments is never displayed.
|
465 |
|
466 |
## Resources for Beginners
|
467 |
|
468 |
-
New to Web Development? Or New to GitHub? Glad to see you're here!! :slightly_smiling_face: Websurfx is a pretty simple app, so it should make a good candidate for your first PR. The following articles (which
|
469 |
|
470 |
**Development**
|
471 |
|
@@ -489,7 +489,7 @@ New to Web Development? Or New to GitHub? Glad to see you're here!! :slightly_sm
|
|
489 |
- [A Tutorial on Git](https://piped.kavin.rocks/playlist?list=PL4lTrYcDuAfxAgSefXftJXbhw0qvjfOFo)
|
490 |
- [Git cheat sheet](http://git-cheatsheet.com/)
|
491 |
|
492 |
-
For Rust, CSS, JS,
|
493 |
|
494 |
## App Info
|
495 |
|
@@ -509,7 +509,7 @@ Styleguides to follow:
|
|
509 |
## Application Structure
|
510 |
|
511 |
> **Important**
|
512 |
-
> We follow the
|
513 |
|
514 |
**Files in the root of the codebase:** `./`
|
515 |
|
@@ -543,21 +543,21 @@ Styleguides to follow:
|
|
543 |
βββ static # The directory containing all the UI handlers.
|
544 |
β βββ cookies.js # Handles the loading of saved cookies.
|
545 |
β βββ error_box.js # Handles the toggling functionality of the error box on the search page.
|
546 |
-
β βββ index.js # Functions to handle the
|
547 |
-
β βββ pagination.js # Functions to handle the navigation between previous and next page in the search page.
|
548 |
-
β βββ search_area_options.js # Changes the search options under the search bar in the search page according to the safe search level set using the
|
549 |
-
β βββ settings.js # Handles the
|
550 |
β βββ colorschemes # A folder containing all the popular colorscheme files as CSS files.
|
551 |
β βββ themes # A folder containing all the popular theme files as CSS files.
|
552 |
βββ templates # Folder containing all the template files for the different pages on the website.
|
553 |
-
βββ 404.html # A 404
|
554 |
βββ about.html # An about page template.
|
555 |
βββ bar.html # A template for the search bar.
|
556 |
βββ cookies_tab.html # A template for the cookies tab for the settings page.
|
557 |
βββ engines_tab.html # A template for the engines tab for the settings page.
|
558 |
-
βββ footer.html # A footer template for all
|
559 |
βββ general_tab.html # A template for the general tab for the settings page.
|
560 |
-
βββ header.html # A header template for all
|
561 |
βββ index.html # A home page template.
|
562 |
βββ navbar.html # A navbar template for the header template.
|
563 |
βββ search.html # A search page template.
|
@@ -572,7 +572,7 @@ Styleguides to follow:
|
|
572 |
./src/
|
573 |
βββ lib.rs # A library file for the rust project.
|
574 |
βββ bin # A folder containing the source code that would produce the binary file when compiled.
|
575 |
-
β βββ websurfx.rs # A file that would be compiled into binary file.
|
576 |
βββ cache # A folder that contains code to handle the caching functionality of the search engine.
|
577 |
β βββ cacher.rs # Handles the different caching features.
|
578 |
β βββ error.rs # Provides custom error messages for different types of caches and their related errors.
|
@@ -587,7 +587,7 @@ Styleguides to follow:
|
|
587 |
β βββ mod.rs # A module file for the rust project.
|
588 |
β βββ search_result_parser.rs # Provides helper function to help ease the process of defining different result selection selectors.
|
589 |
β βββ searx.rs # Provides code to fetch and remove unnecessary or waste results from the fetched results from the searx engine.
|
590 |
-
βββ handler # A folder that provides helper code to provide a proper path to the public (theme) folder, config file, blocklist file and allowlist file based on where they are located.
|
591 |
β βββ mod.rs # A module file for the rust project.
|
592 |
β βββ paths.rs # Provides helper code to handle different paths.
|
593 |
βββ models # A folder that provides different models for the different modules in the backend code.
|
@@ -596,10 +596,10 @@ Styleguides to follow:
|
|
596 |
β βββ mod.rs # A module file for the rust project.
|
597 |
β βββ parser_models.rs # Provides different models (enums, structs) for handling and standardizing different parts in the "config" module code.
|
598 |
β βββ server_models.rs # Provides different models (enums, structs) for handling and standardizing different parts in the "server" module code.
|
599 |
-
βββ results # A folder that provides code to handle the
|
600 |
-
β βββ aggregator.rs # Provides code aggregate and
|
601 |
β βββ mod.rs # A module file for the rust project.
|
602 |
-
β βββ user_agent.rs # Provides a helper function to
|
603 |
βββ server # A folder that holds code to handle the routes for the search engine website.
|
604 |
βββ mod.rs # A module file for the rust project.
|
605 |
βββ router.rs # Provides functions to handle the different routes on the website.
|
@@ -612,19 +612,19 @@ Styleguides to follow:
|
|
612 |
|
613 |
### Performance - Lighthouse
|
614 |
|
615 |
-
The easiest method of checking performance is to use Chromium's
|
616 |
|
617 |
## Notes
|
618 |
|
619 |
### Known warnings
|
620 |
|
621 |
-
When running the build command, a warning
|
622 |
|
623 |
```shell
|
624 |
warning: the following packages contain code that will be rejected by a future version of Rust: html5ever v0.23.0
|
625 |
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 2`
|
626 |
```
|
627 |
|
628 |
-
|
629 |
|
630 |
[β¬
οΈ Go back to Home](./README.md)
|
|
|
1 |
# Developing
|
2 |
|
3 |
+
This page of the docs outlines how to get **Websurfx** up and running in a development environment, and outlines the common workflow, different ways to work on the project, a high-level overview of how the project works, project structure, and the best practices that should be followed when working on the project.
|
4 |
|
5 |
<details>
|
6 |
<summary><b>Table of Contents</b></summary>
|
|
|
30 |
|
31 |
## Setting up the Development Environment
|
32 |
|
33 |
+
By default, we provide four different ways to work on the project. These are as follows:
|
34 |
|
35 |
- [Local Development](#local-development-)
|
36 |
- [Gitpod](#gitpod-)
|
37 |
- [NixOS Dev Shell using Nix Flake](#nixos-dev-shell-using-nix-flake-)
|
38 |
- [Local Development with Docker Compose](#local-development-with-docker-compose-)
|
39 |
|
40 |
+
The different methods are explained in depth below.
|
41 |
|
42 |
### Local Development
|
43 |
|
44 |
+
This section covers how to set up the project for development on your local machine (bare metal).
|
45 |
|
46 |
#### Prerequisites
|
47 |
|
48 |
Before you start working on the project. You will need the following packages installed on your system:
|
49 |
|
50 |
+
- The latest version of `cargo` installed on your system which is required to manage building and running the project. The installation instructions for this can be found [here](https://doc.rust-lang.org/cargo/getting-started/installation.html).
|
51 |
+
- The latest version of `npm` installed on your system which is required to allow the installation of other tools necessary for the project. The installation for this can be found [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
|
52 |
+
- The latest version of `redis` installed on your system which will be used to avoid introducing unexpected issues when working on the project. The installation for this can be found [here](https://redis.io/docs/getting-started/installation/).
|
53 |
+
- The latest version of `stylelint` should be installed on your system which will be used by the pre-commit checks to lint the code before a commit can be made to ensure better code quality. Before you install `stylelint` on your system, make sure you have `npm` installed on your system. To install `stylelint` run the following command:
|
54 |
|
55 |
```shell
|
56 |
$ npm i -g stylelint
|
57 |
```
|
58 |
|
59 |
> **Note**
|
60 |
+
> In the above command the dollar sign(**$**) refers to running the command in privileged mode by using utilities `sudo`, `doas`, `pkgexec`, or any other privileged access methods.
|
61 |
|
62 |
+
- `Cargo-watch` installed on your system which will allow you to auto-build the project when any checks occur in the source code files in the codebase (`websurfx` directory). Before you install `cargo-watch` on your system, make sure you have `cargo` installed on your system. To install `cargo-watch` run the following command:
|
63 |
|
64 |
```shell
|
65 |
cargo install cargo-watch
|
66 |
```
|
67 |
|
68 |
+
- `Git` installed on your system. The installation instructions for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
|
69 |
+
- Finally, The latest version of `Docker` is installed on your system which will be used to avoid introducing unexpected issues when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
|
70 |
|
71 |
> **Note**
|
72 |
+
> For **rolling release Linux distributions (distros)**, the above-mentioned required packages except for `stylelint` and `cargo-watch` can also be installed via the distro-specific package manager.
|
73 |
>
|
74 |
> **For Example:**
|
75 |
>
|
76 |
+
> On `arch linux` the following packages can be installed by following the link to the installation instructions provided below:
|
77 |
>
|
78 |
> - `Cargo`: https://wiki.archlinux.org/title/rust
|
79 |
> - `Npm`: https://wiki.archlinux.org/title/Node.js
|
|
|
81 |
> - `Git`: https://wiki.archlinux.org/title/git
|
82 |
> - `Docker`: https://wiki.archlinux.org/title/docker
|
83 |
>
|
84 |
+
> But we do not recommend this method for **stable release Linux distros** as they tend to not provide very up-to-date versions of the required packages.
|
85 |
|
86 |
#### Setting up Pre-commit Checks
|
87 |
|
88 |
+
Before you set `pre-commit` checks, you will first need to clone **your fork of the project** and navigate into the cloned repository by running the following command:
|
89 |
|
90 |
```shell
|
91 |
git clone https://github.com/<your_github_username>/websurfx.git
|
92 |
cd websurfx
|
93 |
```
|
94 |
|
95 |
+
Once you have finished running the above commands then run the following command to set the `pre-commit` checks:
|
96 |
|
97 |
```shell
|
98 |
cargo test
|
99 |
```
|
100 |
|
101 |
+
By running the above-mentioned command, it will automatically set up all the pre-commit checks in the project.
|
102 |
|
103 |
#### Running the Project
|
104 |
|
|
|
108 |
cargo watch -q -x "run" -w "."
|
109 |
```
|
110 |
|
111 |
+
This will compile the app by default with the **In-Memory caching** feature. To compile, run, and test the app with other features follow the build options listed below:
|
112 |
|
113 |
##### Hybrid Cache
|
114 |
|
|
|
140 |
> redis-server --port 8082 &
|
141 |
> ```
|
142 |
|
143 |
+
Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to be rebuilt.
|
144 |
|
145 |
> For more info on all the project commands. See: [**Project Commands**](#project-commands-)
|
146 |
|
147 |
### Gitpod
|
148 |
|
149 |
+
This section covers how to use and set up the Gitpod development environment for working on the project.
|
150 |
|
151 |
> **Note**
|
152 |
> By default the project only supports the Vscode **IDE/Editor** for Gitpod.
|
|
|
167 |
|
168 |
The project by default provides a set of pre-installed plugins for gitpod which is done to improve productivity and efficiency while working on the project. Also to make working on the project more fun and engaging which can be customized from within the `Gitpod` instance.
|
169 |
|
170 |
+
The list of all the pre-installed plugins are listed below:
|
171 |
|
172 |
**Productivity**
|
173 |
|
|
|
191 |
- [Test Explorer UI](https://open-vsx.org/extension/hbenl/vscode-test-explorer): Provides a way to run any test from a convenient sidebar.
|
192 |
- [GitLens](https://open-vsx.org/extension/eamodio/gitlens): Provides a better and more efficient way to manage common git workflows.
|
193 |
|
194 |
+
> Optionally, if you prefer a more keyboard-centric workflow then we would recommend using the following extension:
|
195 |
>
|
196 |
> - [VSCode Neovim](https://open-vsx.org/extension/asvetliakov/vscode-neovim): Provides complete vim emulation for vscode.
|
197 |
|
|
|
204 |
|
205 |
### NixOS Dev Shell using Nix Flake
|
206 |
|
207 |
+
This section covers how to setup the project for development using the `NixOS dev-shell`.
|
208 |
|
209 |
#### Pre Setup Requirements
|
210 |
|
211 |
Before you start working on the project. You will need the following packages installed on your system:
|
212 |
|
213 |
+
- `Git` installed on your system. The installation instructions for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
|
214 |
+
- Finally, The latest version of `Docker` is installed on your system which will be used to avoid introducing unexpected issues when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
|
215 |
|
216 |
+
> Optionally, On `NixOS` the above-mentioned required packages except for `stylelint` and `cargo-watch` could also be installed by following the link to the installation instructions provided below:
|
217 |
>
|
218 |
> - `Git`: https://search.nixos.org/packages?channel=23.05&show=git&from=0&size=50&sort=relevance&type=packages&query=git
|
219 |
> - `Docker`: https://search.nixos.org/packages?channel=23.05&show=docker&from=0&size=50&sort=relevance&type=packages&query=docker
|
|
|
233 |
nix develop
|
234 |
```
|
235 |
|
236 |
+
Once you have finished running the above commands then run the following command to setup the `pre-commit` checks:
|
237 |
|
238 |
```shell
|
239 |
cargo test
|
240 |
```
|
241 |
|
242 |
+
By running the above-mentioned command, it will automatically set up all the pre-commit checks in the project.
|
243 |
|
244 |
#### Post Setup Requirements
|
245 |
|
246 |
+
After you have done setting up pre-commit checks, then you may need to fulfill a few more requirements to finish setting up the development environment with `NixOS dev-shell`. These include:
|
247 |
|
248 |
+
- `Cargo-watch` installed on your system which will allow you to auto-build the project when any checks occur in the source code files in the codebase (`websurfx` directory). Before you install `cargo-watch` on your system, make sure you have `cargo` installed on your system. To install `cargo-watch` run the following command:
|
249 |
|
250 |
```shell
|
251 |
cargo install cargo-watch
|
|
|
259 |
cargo watch -q -x "run" -w "."
|
260 |
```
|
261 |
|
262 |
+
This will compile the app by default with the **In-Memory caching** feature. To compile, run, and test the app with other features follow the build options listed below:
|
263 |
|
264 |
##### Hybrid Cache
|
265 |
|
|
|
285 |
cargo watch -q -x "run --no-default-features --features redis-cache" -w .
|
286 |
```
|
287 |
|
288 |
+
> Optionally, If you have build and run the app with the `Redis cache`or `Hybrid cache` feature (as mentioned above) then you will need to start the redis server alongside the app which can be done by running the following command:
|
289 |
>
|
290 |
> ```shell
|
291 |
> redis-server --port 8082 &
|
292 |
> ```
|
293 |
|
294 |
+
Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to be rebuilt.
|
295 |
|
296 |
### Local Development with Docker Compose
|
297 |
|
298 |
+
This section covers how to set up the project for development on your local machine (bare metal) using `docker compose`.
|
299 |
|
300 |
#### Prerequisites
|
301 |
|
302 |
Before you start working on the project. You will need the following packages installed on your system:
|
303 |
|
304 |
+
- The latest version of `cargo` installed on your system which is required to manage the building and running the project. The installation instructions for this can be found [here](https://doc.rust-lang.org/cargo/getting-started/installation.html).
|
305 |
+
- The latest version of `npm` installed on your system which is required to allow the installation of other tools necessary for the project. The installation for this can be found [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
|
306 |
+
- The latest version of `stylelint` should be installed on your system which will be used by the pre-commit checks to lint the code before a commit can be made to ensure better code quality. Before you install `stylelint` on your system, make sure you have `npm` installed on your system. To install `stylelint` run the following command:
|
307 |
|
308 |
```shell
|
309 |
$ npm i -g stylelint
|
310 |
```
|
311 |
|
312 |
> **Note**
|
313 |
+
> In the above command the dollar sign(**$**) refers to running the command in privileged mode by using utilities `sudo`, `doas`, `pkgexec`, or any other privileged access methods.
|
314 |
|
315 |
+
- `Git` installed on your system. The installation instructions for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
|
316 |
+
- Finally, The latest version of `Docker` is installed on your system which will be used to avoid introducing unexpected issues when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
|
317 |
|
318 |
> **Note**
|
319 |
+
> For **rolling release Linux distributions (distros)**, the above-mentioned all required packages can also be installed via the distro-specific package manager.
|
320 |
>
|
321 |
> **For Example:**
|
322 |
>
|
323 |
+
> On `arch linux` the following packages can be installed by following the link to the installation instructions provided below:
|
324 |
>
|
325 |
> - `Cargo`: https://wiki.archlinux.org/title/rust
|
326 |
> - `Npm`: https://wiki.archlinux.org/title/Node.js
|
327 |
> - `Git`: https://wiki.archlinux.org/title/git
|
328 |
> - `Docker`: https://wiki.archlinux.org/title/docker
|
329 |
>
|
330 |
+
> But we do not recommend this method for **stable release Linux distros** as they tend to not provide very up-to-date versions of the required packages.
|
331 |
|
332 |
#### Setting up Pre-commit Checks
|
333 |
|
|
|
338 |
cd websurfx
|
339 |
```
|
340 |
|
341 |
+
Once you have finished running the above commands then run the following command to setup the `pre-commit` checks:
|
342 |
|
343 |
```shell
|
344 |
cargo test
|
345 |
```
|
346 |
|
347 |
+
By running the above-mentioned command, it will automatically set up all the pre-commit checks in the project.
|
348 |
|
349 |
#### Running the Project
|
350 |
|
|
|
355 |
```
|
356 |
|
357 |
> **Note**
|
358 |
+
> In the above command the dollar sign(**$**) refers to running the command in privileged mode by using utilities `sudo`, `doas`, `pkgexec`, or any other privileged access methods.
|
359 |
|
360 |
+
Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to be rebuilt.
|
361 |
|
362 |
### Project Commands
|
363 |
|
|
|
383 |
|
384 |
### Environment Variables
|
385 |
|
386 |
+
All environment variables are optional. Currently, there are not many environment variables used, as most of the user preferences are stored under the `websurfx` folder (located under the codebase (`websurfx` directory)) in the `config.lua` file.
|
387 |
|
388 |
+
The list of all the available environment variables are listed below:
|
389 |
|
390 |
+
- `PKG_ENV`: Sets the logging level for the app to **Trace** which can be useful for better debugging of the app. These environment variables accept two values `dev` or `prod` as strings.
|
391 |
+
- `RUST_BACKTRACE`: Rust-specific environment variable useful for getting more elaborate error messages with an error stack to better diagnose the issue. This environment variable accepts three values `0` (off), `1` (on), and `full` (for long error stack to being printed out).
|
392 |
|
393 |
## Git Strategy
|
394 |
|
|
|
396 |
|
397 |
Like most Git repos, we are following the [Github Flow](https://guides.github.com/introduction/flow) standard.
|
398 |
|
399 |
+
1. Create a branch (or fork if you don't have write access)
|
400 |
2. Code some awesome stuff π§βπ»
|
401 |
+
3. Add, commit, and push your changes to your branch/ fork
|
402 |
4. Head over to GitHub and create a Pull Request
|
403 |
5. Fill in the required sections in the template, and hit submit
|
404 |
6. Follow up with any reviews on your code
|
|
|
411 |
|
412 |
### Commit Emojis
|
413 |
|
414 |
+
Using a single emoji at the start of each commit message, issue title, and pull request title, to indicate the type of task, makes the commit ledger, issue, and pull request easier to understand, it looks cool.
|
415 |
|
416 |
+
- π¨ `:art:` - Improve the structure/format of the code.
|
417 |
- β‘οΈ `:zap:` - Improve performance.
|
418 |
- π₯ `:fire:` - Remove code or files.
|
419 |
- π `:bug:` - Fix a bug.
|
|
|
426 |
- β
`:white_check_mark:` - Add, update, or pass tests.
|
427 |
- ποΈ `:lock:` - Fix security issues.
|
428 |
- π `:bookmark:` - Make a Release or Version tag.
|
429 |
+
- π¨ `:rotating_light:` - Fix compiler/linter warnings.
|
430 |
- π§ `:construction:` - Work in progress.
|
431 |
- β¬οΈ `:arrow_up:` - Upgrade dependencies.
|
432 |
+
- π· `:construction_worker:` - Add or update the CI build system.
|
433 |
- β»οΈ `:recycle:` - Refactor code.
|
434 |
- π©Ή `:adhesive_bandage:` - Simple fix for a non-critical issue.
|
435 |
- π§ `:wrench:` - Add or update configuration files.
|
436 |
- π± `:bento:` - Add or update assets.
|
437 |
+
- ποΈ `:card_file_box:` - Perform database schema-related changes.
|
438 |
- βοΈ `:pencil2:` - Fix typos.
|
439 |
- π `:globe_with_meridians:` - Internationalization and translations.
|
440 |
|
|
|
446 |
|
447 |
For a pull request to be merged, it must:
|
448 |
|
449 |
+
- The build, lint, and tests (run by GH actions) must pass
|
450 |
- There must not be any merge conflicts
|
451 |
|
452 |
When you submit your pull request, include the required info, by filling out the pull request template. Including:
|
453 |
|
454 |
- A brief description of your changes.
|
455 |
- The issue or ticket number (if applicable).
|
456 |
+
- For UI-related updates include a screenshot.
|
457 |
+
- If any dependencies were added, explain why it was needed, and state the cost. associated, and confirm it does not introduce any security, privacy, or speed issues
|
458 |
+
- Optionally, provide a checklist of all the changes that were included in the pull request.
|
459 |
|
460 |
> **Important**
|
461 |
+
> Make sure to fill all the required/mandatory sections of the pull request as filling them helps us distinguish between spam pull requests and legitimate pull requests.
|
462 |
|
463 |
> **Note**
|
464 |
> The pull request template contains comments in the following form `<!-- -->` which are used to provide a guide on what should be provided under each heading of the template. These comments are never rendered when the pull request is either created or updated and hence anything provided in such comments is never displayed.
|
465 |
|
466 |
## Resources for Beginners
|
467 |
|
468 |
+
New to Web Development? Or New to GitHub? Glad to see you're here!! :slightly_smiling_face: Websurfx is a pretty simple app, so it should make a good candidate for your first PR. The following articles (which have been divided into parts for convenience) should point you in the right direction for getting up to speed with the technologies used in this project:
|
469 |
|
470 |
**Development**
|
471 |
|
|
|
489 |
- [A Tutorial on Git](https://piped.kavin.rocks/playlist?list=PL4lTrYcDuAfxAgSefXftJXbhw0qvjfOFo)
|
490 |
- [Git cheat sheet](http://git-cheatsheet.com/)
|
491 |
|
492 |
+
For Rust, CSS, JS, HTML, Git, and Docker- you'll need an IDE (e.g. [VSCode](https://code.visualstudio.com/) or [Neovim](https://neovim.io/) and a terminal (Windows users may find [WSL](https://docs.microsoft.com/en-us/windows/wsl/) more convenient).
|
493 |
|
494 |
## App Info
|
495 |
|
|
|
509 |
## Application Structure
|
510 |
|
511 |
> **Important**
|
512 |
+
> We follow the Unix style naming conventions for all the files and folders in the project (except for all files under the `themes` and `colorschemes` folder in the frontend's source code which requires that the names of the files and folders should be in lowercase and the words be separated with a hyphen.) which includes the name of the files and folders should be in lowercase and every word should be separated with an underscore.
|
513 |
|
514 |
**Files in the root of the codebase:** `./`
|
515 |
|
|
|
543 |
βββ static # The directory containing all the UI handlers.
|
544 |
β βββ cookies.js # Handles the loading of saved cookies.
|
545 |
β βββ error_box.js # Handles the toggling functionality of the error box on the search page.
|
546 |
+
β βββ index.js # Functions to handle the search functionality of the search bar.
|
547 |
+
β βββ pagination.js # Functions to handle the navigation between the previous and next page in the search page.
|
548 |
+
β βββ search_area_options.js # Changes the search options under the search bar in the search page according to the safe search level set using the URL safesearch parameter.
|
549 |
+
β βββ settings.js # Handles the settings and saving of all the settings page options as a cookie.
|
550 |
β βββ colorschemes # A folder containing all the popular colorscheme files as CSS files.
|
551 |
β βββ themes # A folder containing all the popular theme files as CSS files.
|
552 |
βββ templates # Folder containing all the template files for the different pages on the website.
|
553 |
+
βββ 404.html # A 404-page template.
|
554 |
βββ about.html # An about page template.
|
555 |
βββ bar.html # A template for the search bar.
|
556 |
βββ cookies_tab.html # A template for the cookies tab for the settings page.
|
557 |
βββ engines_tab.html # A template for the engines tab for the settings page.
|
558 |
+
βββ footer.html # A footer template for all pages.
|
559 |
βββ general_tab.html # A template for the general tab for the settings page.
|
560 |
+
βββ header.html # A header template for all pages.
|
561 |
βββ index.html # A home page template.
|
562 |
βββ navbar.html # A navbar template for the header template.
|
563 |
βββ search.html # A search page template.
|
|
|
572 |
./src/
|
573 |
βββ lib.rs # A library file for the rust project.
|
574 |
βββ bin # A folder containing the source code that would produce the binary file when compiled.
|
575 |
+
β βββ websurfx.rs # A file that would be compiled into a binary file.
|
576 |
βββ cache # A folder that contains code to handle the caching functionality of the search engine.
|
577 |
β βββ cacher.rs # Handles the different caching features.
|
578 |
β βββ error.rs # Provides custom error messages for different types of caches and their related errors.
|
|
|
587 |
β βββ mod.rs # A module file for the rust project.
|
588 |
β βββ search_result_parser.rs # Provides helper function to help ease the process of defining different result selection selectors.
|
589 |
β βββ searx.rs # Provides code to fetch and remove unnecessary or waste results from the fetched results from the searx engine.
|
590 |
+
βββ handler # A folder that provides helper code to provide a proper path to the public (theme) folder, config file, blocklist file, and allowlist file based on where they are located.
|
591 |
β βββ mod.rs # A module file for the rust project.
|
592 |
β βββ paths.rs # Provides helper code to handle different paths.
|
593 |
βββ models # A folder that provides different models for the different modules in the backend code.
|
|
|
596 |
β βββ mod.rs # A module file for the rust project.
|
597 |
β βββ parser_models.rs # Provides different models (enums, structs) for handling and standardizing different parts in the "config" module code.
|
598 |
β βββ server_models.rs # Provides different models (enums, structs) for handling and standardizing different parts in the "server" module code.
|
599 |
+
βββ results # A folder that provides code to handle the fetching and aggregating of results from the upstream search engines.
|
600 |
+
β βββ aggregator.rs # Provides code aggregate and fetches results from the upstream engines.
|
601 |
β βββ mod.rs # A module file for the rust project.
|
602 |
+
β βββ user_agent.rs # Provides a helper function to allow random user agents to pass in the server request code to improve user privacy and avoiding detected as a bot.
|
603 |
βββ server # A folder that holds code to handle the routes for the search engine website.
|
604 |
βββ mod.rs # A module file for the rust project.
|
605 |
βββ router.rs # Provides functions to handle the different routes on the website.
|
|
|
612 |
|
613 |
### Performance - Lighthouse
|
614 |
|
615 |
+
The easiest method of checking performance is to use Chromium's built-in auditing tool, Lighthouse. To run the test, open Developer Tools (usually F12) --> Lighthouse and click on the 'Generate Report' button at the bottom.
|
616 |
|
617 |
## Notes
|
618 |
|
619 |
### Known warnings
|
620 |
|
621 |
+
When running the build command, a warning appears. This is not an error and does not affect the security or performance of the application. They will be addressed soon in a future update.
|
622 |
|
623 |
```shell
|
624 |
warning: the following packages contain code that will be rejected by a future version of Rust: html5ever v0.23.0
|
625 |
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 2`
|
626 |
```
|
627 |
|
628 |
+
This warning just means that any dependencies or code using the `html5ever` code would be deprecated and rejected in future versions of the Rust language. So right now these dependencies can be used as these have not happened yet.
|
629 |
|
630 |
[β¬
οΈ Go back to Home](./README.md)
|