Shivam Kumar commited on
Commit
344e641
β€’
1 Parent(s): 82802ba

πŸ“ Typo correction and Grammatical correction in the `developing` documentation page (#360)

Browse files
Files changed (1) hide show
  1. docs/developing.md +82 -82
docs/developing.md CHANGED
@@ -1,6 +1,6 @@
1
  # Developing
2
 
3
- This page of the docs outlines how to get **Websurfx** up and running in a development environment, and outlines the common worflow, different ways to work on the project, a high level overview of how the project works, project structure and the best pratices that should be followed when working on the project.
4
 
5
  <details>
6
  <summary><b>Table of Contents</b></summary>
@@ -30,50 +30,50 @@ This page of the docs outlines how to get **Websurfx** up and running in a devel
30
 
31
  ## Setting up the Development Environment
32
 
33
- By default, we provide four different ways to work on the project. Which are as follows:
34
 
35
  - [Local Development](#local-development-)
36
  - [Gitpod](#gitpod-)
37
  - [NixOS Dev Shell using Nix Flake](#nixos-dev-shell-using-nix-flake-)
38
  - [Local Development with Docker Compose](#local-development-with-docker-compose-)
39
 
40
- The diferent methods are explained in depth below.
41
 
42
  ### Local Development
43
 
44
- This section covers how to setup the project for development on your local machine (bare metal).
45
 
46
  #### Prerequisites
47
 
48
  Before you start working on the project. You will need the following packages installed on your system:
49
 
50
- - A latest version of `cargo` installed on your system which is required to manage building and running the project. The installation instruction for this can be found [here](https://doc.rust-lang.org/cargo/getting-started/installation.html).
51
- - A latest version of `npm` installed on your system which is required to allow the installation of other tools necessary for the project. The installation for this can be found [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
52
- - A latest version of `redis` installed on your system which will be used to avoid introducing unexpected issue when working on the project. The installation for this can be found [here](https://redis.io/docs/getting-started/installation/).
53
- - A latest version of `stylelint` should be installed on your system which will be used by the pre-commit checks to lint the code before a commit can be made to ensure better code quality. Before you install `stylelint` on your system, make sure you have `npm` installed on you system. To install `stylelint` run the following command:
54
 
55
  ```shell
56
  $ npm i -g stylelint
57
  ```
58
 
59
  > **Note**
60
- > In the above command the dollar sign(**$**) refers to running the command in privilaged mode by using utilities `sudo`, `doas`, `pkgexec` or any other privilage access methods.
61
 
62
- - `Cargo-watch` installed on your system which will allow to auto build the project when any checks occur in the source code files in the codebase (`websurfx` directory). Before you install `cargo-watch` on your system, make sure you have `cargo` installed on your system. To install `cargo-watch` run the following command:
63
 
64
  ```shell
65
  cargo install cargo-watch
66
  ```
67
 
68
- - `Git` installed on your system. The installation instruction for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
69
- - Finally, A latest version of `Docker` installed on your system which will be used to avoid introducing unexpected issue when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
70
 
71
  > **Note**
72
- > For **rolling release linux distributions (distros)**, the above mentioned all required packages except for `stylelint` and `cargo-watch` can also be installed via the distro specific package manager.
73
  >
74
  > **For Example:**
75
  >
76
- > On `arch linux` the following packages could be installed by following link to the installation instructions provided down below:
77
  >
78
  > - `Cargo`: https://wiki.archlinux.org/title/rust
79
  > - `Npm`: https://wiki.archlinux.org/title/Node.js
@@ -81,24 +81,24 @@ cargo install cargo-watch
81
  > - `Git`: https://wiki.archlinux.org/title/git
82
  > - `Docker`: https://wiki.archlinux.org/title/docker
83
  >
84
- > But we do not recommend this method for **stable release linux distros** as they tend to not provide very up to date versions of the required packages.
85
 
86
  #### Setting up Pre-commit Checks
87
 
88
- Before you setup `pre-commit` checks, you will first need to clone **your fork of the project** and navigate into the cloned repository by running the following command:
89
 
90
  ```shell
91
  git clone https://github.com/<your_github_username>/websurfx.git
92
  cd websurfx
93
  ```
94
 
95
- Once you have done finished running the above commands then run the following command to setup the `pre-commit` checks:
96
 
97
  ```shell
98
  cargo test
99
  ```
100
 
101
- By the running the above mentioned command, it will automatically setup all the pre-commit checks in the project.
102
 
103
  #### Running the Project
104
 
@@ -108,7 +108,7 @@ If you have followed the above section then you should have a cloned repository
108
  cargo watch -q -x "run" -w "."
109
  ```
110
 
111
- This will compile the app by default with the **In-Memory caching** feature. To compile, run and test the app with other features then follow the build options listed below:
112
 
113
  ##### Hybrid Cache
114
 
@@ -140,13 +140,13 @@ cargo watch -q -x "run --no-default-features --features redis-cache" -w .
140
  > redis-server --port 8082 &
141
  > ```
142
 
143
- Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to being rebuilt.
144
 
145
  > For more info on all the project commands. See: [**Project Commands**](#project-commands-)
146
 
147
  ### Gitpod
148
 
149
- This section covers on how to use and setup the Gitpod development environment for working on the project.
150
 
151
  > **Note**
152
  > By default the project only supports the Vscode **IDE/Editor** for Gitpod.
@@ -167,7 +167,7 @@ https://gitpod.io/#https://github.com/<your_github_username>/websurfx
167
 
168
  The project by default provides a set of pre-installed plugins for gitpod which is done to improve productivity and efficiency while working on the project. Also to make working on the project more fun and engaging which can be customized from within the `Gitpod` instance.
169
 
170
- The list of all the pre-installed plugins are listed down below:
171
 
172
  **Productivity**
173
 
@@ -191,7 +191,7 @@ The list of all the pre-installed plugins are listed down below:
191
  - [Test Explorer UI](https://open-vsx.org/extension/hbenl/vscode-test-explorer): Provides a way to run any test from a convenient sidebar.
192
  - [GitLens](https://open-vsx.org/extension/eamodio/gitlens): Provides a better and more efficient way to manage common git workflows.
193
 
194
- > Optionally, if you prefer more keyboard centric workflow then we would recommend using the following extension:
195
  >
196
  > - [VSCode Neovim](https://open-vsx.org/extension/asvetliakov/vscode-neovim): Provides complete vim emulation for vscode.
197
 
@@ -204,16 +204,16 @@ The list of all the pre-installed plugins are listed down below:
204
 
205
  ### NixOS Dev Shell using Nix Flake
206
 
207
- This section covers on how to setup the project for development using the `NixOS dev-shell`.
208
 
209
  #### Pre Setup Requirements
210
 
211
  Before you start working on the project. You will need the following packages installed on your system:
212
 
213
- - `Git` installed on your system. The installation instruction for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
214
- - Finally, A latest version of `Docker` installed on your system which will be used to avoid introducing unexpected issue when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
215
 
216
- > Optionally, On `NixOS` the above mentioned all required packages except for `stylelint` and `cargo-watch` could also be installed by following link to the installation instructions provided down below:
217
  >
218
  > - `Git`: https://search.nixos.org/packages?channel=23.05&show=git&from=0&size=50&sort=relevance&type=packages&query=git
219
  > - `Docker`: https://search.nixos.org/packages?channel=23.05&show=docker&from=0&size=50&sort=relevance&type=packages&query=docker
@@ -233,19 +233,19 @@ Then run the following command to setup the `NixOS dev-shell`:
233
  nix develop
234
  ```
235
 
236
- Once you have done finished running the above commands then run the following command to setup the `pre-commit` checks:
237
 
238
  ```shell
239
  cargo test
240
  ```
241
 
242
- By the running the above mentioned command, it will automatically setup all the pre-commit checks in the project.
243
 
244
  #### Post Setup Requirements
245
 
246
- After you have done setting up pre-commit checks, then you may need to fullfill a few more requirements to finish setting up the development environment with `NixOS dev-shell`. These include:
247
 
248
- - `Cargo-watch` installed on your system which will allow to auto build the project when any checks occur in the source code files in the codebase (`websurfx` directory). Before you install `cargo-watch` on your system, make sure you have `cargo` installed on your system. To install `cargo-watch` run the following command:
249
 
250
  ```shell
251
  cargo install cargo-watch
@@ -259,7 +259,7 @@ If you have followed the above section then you should now be inside a `dev-shel
259
  cargo watch -q -x "run" -w "."
260
  ```
261
 
262
- This will compile the app by default with the **In-Memory caching** feature. To compile, run and test the app with other features then follow the build options listed below:
263
 
264
  ##### Hybrid Cache
265
 
@@ -285,49 +285,49 @@ To build the search engine with the `Redis caching` feature. Run the following c
285
  cargo watch -q -x "run --no-default-features --features redis-cache" -w .
286
  ```
287
 
288
- > Optionally, If you have build and run the app with the `Redis cache`or `Hybrid cache` feature (as mentioned above) then you will need to start the redis server alongside the app which can be done so by running the following command:
289
  >
290
  > ```shell
291
  > redis-server --port 8082 &
292
  > ```
293
 
294
- Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to being rebuilt.
295
 
296
  ### Local Development with Docker Compose
297
 
298
- This section covers how to setup the project for development on your local machine (bare metal) using `docker compose`.
299
 
300
  #### Prerequisites
301
 
302
  Before you start working on the project. You will need the following packages installed on your system:
303
 
304
- - A latest version of `cargo` installed on your system which is required to manage building and running the project. The installation instruction for this can be found [here](https://doc.rust-lang.org/cargo/getting-started/installation.html).
305
- - A latest version of `npm` installed on your system which is required to allow the installation of other tools necessary for the project. The installation for this can be found [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
306
- - A latest version of `stylelint` should be installed on your system which will be used by the pre-commit checks to lint the code before a commit can be made to ensure better code quality. Before you install `stylelint` on your system, make sure you have `npm` installed on you system. To install `stylelint` run the following command:
307
 
308
  ```shell
309
  $ npm i -g stylelint
310
  ```
311
 
312
  > **Note**
313
- > In the above command the dollar sign(**$**) refers to running the command in privilaged mode by using utilities `sudo`, `doas`, `pkgexec` or any other privilage access methods.
314
 
315
- - `Git` installed on your system. The installation instruction for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
316
- - Finally, A latest version of `Docker` installed on your system which will be used to avoid introducing unexpected issue when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
317
 
318
  > **Note**
319
- > For **rolling release linux distributions (distros)**, the above mentioned all required packages can also be installed via the distro specific package manager.
320
  >
321
  > **For Example:**
322
  >
323
- > On `arch linux` the following packages could be installed by following link to the installation instructions provided down below:
324
  >
325
  > - `Cargo`: https://wiki.archlinux.org/title/rust
326
  > - `Npm`: https://wiki.archlinux.org/title/Node.js
327
  > - `Git`: https://wiki.archlinux.org/title/git
328
  > - `Docker`: https://wiki.archlinux.org/title/docker
329
  >
330
- > But we do not recommend this method for **stable release linux distros** as they tend to not provide very up to date versions of the required packages.
331
 
332
  #### Setting up Pre-commit Checks
333
 
@@ -338,13 +338,13 @@ git clone https://github.com/<your_github_username>/websurfx.git
338
  cd websurfx
339
  ```
340
 
341
- Once you have done finished running the above commands then run the following command to setup the `pre-commit` checks:
342
 
343
  ```shell
344
  cargo test
345
  ```
346
 
347
- By the running the above mentioned command, it will automatically setup all the pre-commit checks in the project.
348
 
349
  #### Running the Project
350
 
@@ -355,9 +355,9 @@ $ docker compose -f dev.docker-compose.yml up
355
  ```
356
 
357
  > **Note**
358
- > In the above command the dollar sign(**$**) refers to running the command in privilaged mode by using utilities `sudo`, `doas`, `pkgexec` or any other privilage access methods.
359
 
360
- Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to being rebuilt.
361
 
362
  ### Project Commands
363
 
@@ -383,12 +383,12 @@ Once you have finished running the above command, Websurfx should now be served
383
 
384
  ### Environment Variables
385
 
386
- All environment variables are optional. Currently there are not many environment variables used, as most of the user preferences are stored under the `websurfx` folder (located under the codebase (`websurfx` directory)) in the `config.lua` file.
387
 
388
- The list of all the available environment variables are listed down below:
389
 
390
- - `PKG_ENV`: Sets the logging level for the app to **Trace** which can be useful for better debugging of the app. This environment variables accept two values `dev` or `prod` as strings.
391
- - `RUST_BACKTRACE`: Rust specific environment variable useful for getting more elaborate error messages with an error stack to better diagnose the issue. This environment variable accepts three values `0` (off), `1` (on) and `full` (for long error stack to being printed out).
392
 
393
  ## Git Strategy
394
 
@@ -396,9 +396,9 @@ The list of all the available environment variables are listed down below:
396
 
397
  Like most Git repos, we are following the [Github Flow](https://guides.github.com/introduction/flow) standard.
398
 
399
- 1. Create a branch (or fork if you don'd have write access)
400
  2. Code some awesome stuff πŸ§‘β€πŸ’»
401
- 3. Add, commit and push your changes to your branch/ fork
402
  4. Head over to GitHub and create a Pull Request
403
  5. Fill in the required sections in the template, and hit submit
404
  6. Follow up with any reviews on your code
@@ -411,9 +411,9 @@ For example, `FEATURE/420_Awesome-feature` or `FIX/690_login-server-error`
411
 
412
  ### Commit Emojis
413
 
414
- Using a single emoji at the start of each commit message, issue title, and pull request title, to indicate the type task, makes the commit ledger, issue and pull request easier to understand, plus it looks cool.
415
 
416
- - 🎨 `:art:` - Improve structure / format of the code.
417
  - ⚑️ `:zap:` - Improve performance.
418
  - πŸ”₯ `:fire:` - Remove code or files.
419
  - πŸ› `:bug:` - Fix a bug.
@@ -426,15 +426,15 @@ Using a single emoji at the start of each commit message, issue title, and pull
426
  - βœ… `:white_check_mark:` - Add, update, or pass tests.
427
  - πŸ”’οΈ `:lock:` - Fix security issues.
428
  - πŸ”– `:bookmark:` - Make a Release or Version tag.
429
- - 🚨 `:rotating_light:` - Fix compiler / linter warnings.
430
  - 🚧 `:construction:` - Work in progress.
431
  - ⬆️ `:arrow_up:` - Upgrade dependencies.
432
- - πŸ‘· `:construction_worker:` - Add or update CI build system.
433
  - ♻️ `:recycle:` - Refactor code.
434
  - 🩹 `:adhesive_bandage:` - Simple fix for a non-critical issue.
435
  - πŸ”§ `:wrench:` - Add or update configuration files.
436
  - 🍱 `:bento:` - Add or update assets.
437
- - πŸ—ƒοΈ `:card_file_box:` - Perform database schema related changes.
438
  - ✏️ `:pencil2:` - Fix typos.
439
  - 🌐 `:globe_with_meridians:` - Internationalization and translations.
440
 
@@ -446,26 +446,26 @@ Once you've made your changes, and pushed them to your fork or branch, you're re
446
 
447
  For a pull request to be merged, it must:
448
 
449
- - The build, lint and tests (run by GH actions) must pass
450
  - There must not be any merge conflicts
451
 
452
  When you submit your pull request, include the required info, by filling out the pull request template. Including:
453
 
454
  - A brief description of your changes.
455
  - The issue or ticket number (if applicable).
456
- - For UI related updates include a screenshot.
457
- - If any dependencies were added, explain why it was needed, state the cost. associated, and confirm it does not introduce any security, privacy or speed issues
458
- - Optionally, provide a checklist of all the changes that was included in the pull request.
459
 
460
  > **Important**
461
- > Make sure to fill all the required/mandatory sections of the pull request as by filling them it helps us distinguish between spam pull requests and legitimate pull requests.
462
 
463
  > **Note**
464
  > The pull request template contains comments in the following form `<!-- -->` which are used to provide a guide on what should be provided under each heading of the template. These comments are never rendered when the pull request is either created or updated and hence anything provided in such comments is never displayed.
465
 
466
  ## Resources for Beginners
467
 
468
- New to Web Development? Or New to GitHub? Glad to see you're here!! :slightly_smiling_face: Websurfx is a pretty simple app, so it should make a good candidate for your first PR. The following articles (which has been divided into parts for convenience) should point you in the right direction for getting up to speed with the technologies used in this project:
469
 
470
  **Development**
471
 
@@ -489,7 +489,7 @@ New to Web Development? Or New to GitHub? Glad to see you're here!! :slightly_sm
489
  - [A Tutorial on Git](https://piped.kavin.rocks/playlist?list=PL4lTrYcDuAfxAgSefXftJXbhw0qvjfOFo)
490
  - [Git cheat sheet](http://git-cheatsheet.com/)
491
 
492
- For Rust, CSS, JS, Html, Git and Docker- you'll need an IDE (e.g. [VSCode](https://code.visualstudio.com/) or [Neovim](https://neovim.io/) and a terminal (Windows users may find [WSL](https://docs.microsoft.com/en-us/windows/wsl/) more convenient).
493
 
494
  ## App Info
495
 
@@ -509,7 +509,7 @@ Styleguides to follow:
509
  ## Application Structure
510
 
511
  > **Important**
512
- > We follow the unix style naming conventions for all the file and folders in the project (except for all files under the `themes` and `colorschemes` folder in the frontend's source code which requires that the names of the files and folders should be in lowercase and the words be seperated with a hyphen.) which includes the name of the files and folders should be in lowercase and every word should be seperated with an underscore.
513
 
514
  **Files in the root of the codebase:** `./`
515
 
@@ -543,21 +543,21 @@ Styleguides to follow:
543
  β”œβ”€β”€ static # The directory containing all the UI handlers.
544
  β”‚ β”œβ”€β”€ cookies.js # Handles the loading of saved cookies.
545
  β”‚ β”œβ”€β”€ error_box.js # Handles the toggling functionality of the error box on the search page.
546
- β”‚ β”œβ”€β”€ index.js # Functions to handle the searching functionality of the search bar.
547
- β”‚ β”œβ”€β”€ pagination.js # Functions to handle the navigation between previous and next page in the search page.
548
- β”‚ β”œβ”€β”€ search_area_options.js # Changes the search options under the search bar in the search page according to the safe search level set using the url safesearch parameter.
549
- β”‚ β”œβ”€β”€ settings.js # Handles the getting and saving of all the settings page options as a cookie.
550
  β”‚ β”œβ”€β”€ colorschemes # A folder containing all the popular colorscheme files as CSS files.
551
  β”‚ └── themes # A folder containing all the popular theme files as CSS files.
552
  └── templates # Folder containing all the template files for the different pages on the website.
553
- β”œβ”€β”€ 404.html # A 404 page template.
554
  β”œβ”€β”€ about.html # An about page template.
555
  β”œβ”€β”€ bar.html # A template for the search bar.
556
  β”œβ”€β”€ cookies_tab.html # A template for the cookies tab for the settings page.
557
  β”œβ”€β”€ engines_tab.html # A template for the engines tab for the settings page.
558
- β”œβ”€β”€ footer.html # A footer template for all page.
559
  β”œβ”€β”€ general_tab.html # A template for the general tab for the settings page.
560
- β”œβ”€β”€ header.html # A header template for all page.
561
  β”œβ”€β”€ index.html # A home page template.
562
  β”œβ”€β”€ navbar.html # A navbar template for the header template.
563
  β”œβ”€β”€ search.html # A search page template.
@@ -572,7 +572,7 @@ Styleguides to follow:
572
  ./src/
573
  β”œβ”€β”€ lib.rs # A library file for the rust project.
574
  β”œβ”€β”€ bin # A folder containing the source code that would produce the binary file when compiled.
575
- β”‚ └── websurfx.rs # A file that would be compiled into binary file.
576
  β”œβ”€β”€ cache # A folder that contains code to handle the caching functionality of the search engine.
577
  β”‚ β”œβ”€β”€ cacher.rs # Handles the different caching features.
578
  β”‚ β”œβ”€β”€ error.rs # Provides custom error messages for different types of caches and their related errors.
@@ -587,7 +587,7 @@ Styleguides to follow:
587
  β”‚ β”œβ”€β”€ mod.rs # A module file for the rust project.
588
  β”‚ β”œβ”€β”€ search_result_parser.rs # Provides helper function to help ease the process of defining different result selection selectors.
589
  β”‚ └── searx.rs # Provides code to fetch and remove unnecessary or waste results from the fetched results from the searx engine.
590
- β”œβ”€β”€ handler # A folder that provides helper code to provide a proper path to the public (theme) folder, config file, blocklist file and allowlist file based on where they are located.
591
  β”‚ β”œβ”€β”€ mod.rs # A module file for the rust project.
592
  β”‚ └── paths.rs # Provides helper code to handle different paths.
593
  β”œβ”€β”€ models # A folder that provides different models for the different modules in the backend code.
@@ -596,10 +596,10 @@ Styleguides to follow:
596
  β”‚ β”œβ”€β”€ mod.rs # A module file for the rust project.
597
  β”‚ β”œβ”€β”€ parser_models.rs # Provides different models (enums, structs) for handling and standardizing different parts in the "config" module code.
598
  β”‚ └── server_models.rs # Provides different models (enums, structs) for handling and standardizing different parts in the "server" module code.
599
- β”œβ”€β”€ results # A folder that provides code to handle the feching and aggregating of results from the upstream search engines.
600
- β”‚ β”œβ”€β”€ aggregator.rs # Provides code aggregate and fetch results from the upstream engines.
601
  β”‚ β”œβ”€β”€ mod.rs # A module file for the rust project.
602
- β”‚ └── user_agent.rs # Provides a helper function to provide random user-agents to pass in the server request code to improve user privacy and avoiding detected as a bot.
603
  └── server # A folder that holds code to handle the routes for the search engine website.
604
  β”œβ”€β”€ mod.rs # A module file for the rust project.
605
  β”œβ”€β”€ router.rs # Provides functions to handle the different routes on the website.
@@ -612,19 +612,19 @@ Styleguides to follow:
612
 
613
  ### Performance - Lighthouse
614
 
615
- The easiest method of checking performance is to use Chromium's build in auditing tool, Lighthouse. To run the test, open Developer Tools (usually F12) --> Lighthouse and click on the 'Generate Report' button at the bottom.
616
 
617
  ## Notes
618
 
619
  ### Known warnings
620
 
621
- When running the build command, a warning appear. These is not an error, and do not affect the security or performance of the application. They will be addressed soon in a future update.
622
 
623
  ```shell
624
  warning: the following packages contain code that will be rejected by a future version of Rust: html5ever v0.23.0
625
  note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 2`
626
  ```
627
 
628
- These warning just means that any dependencies or code using the `html5ever` code would be deprecated and rejected in the futures versions of the rust lanaguage. So right now these dependencies can be used as these has not happened yet.
629
 
630
  [⬅️ Go back to Home](./README.md)
 
1
  # Developing
2
 
3
+ This page of the docs outlines how to get **Websurfx** up and running in a development environment, and outlines the common workflow, different ways to work on the project, a high-level overview of how the project works, project structure, and the best practices that should be followed when working on the project.
4
 
5
  <details>
6
  <summary><b>Table of Contents</b></summary>
 
30
 
31
  ## Setting up the Development Environment
32
 
33
+ By default, we provide four different ways to work on the project. These are as follows:
34
 
35
  - [Local Development](#local-development-)
36
  - [Gitpod](#gitpod-)
37
  - [NixOS Dev Shell using Nix Flake](#nixos-dev-shell-using-nix-flake-)
38
  - [Local Development with Docker Compose](#local-development-with-docker-compose-)
39
 
40
+ The different methods are explained in depth below.
41
 
42
  ### Local Development
43
 
44
+ This section covers how to set up the project for development on your local machine (bare metal).
45
 
46
  #### Prerequisites
47
 
48
  Before you start working on the project. You will need the following packages installed on your system:
49
 
50
+ - The latest version of `cargo` installed on your system which is required to manage building and running the project. The installation instructions for this can be found [here](https://doc.rust-lang.org/cargo/getting-started/installation.html).
51
+ - The latest version of `npm` installed on your system which is required to allow the installation of other tools necessary for the project. The installation for this can be found [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
52
+ - The latest version of `redis` installed on your system which will be used to avoid introducing unexpected issues when working on the project. The installation for this can be found [here](https://redis.io/docs/getting-started/installation/).
53
+ - The latest version of `stylelint` should be installed on your system which will be used by the pre-commit checks to lint the code before a commit can be made to ensure better code quality. Before you install `stylelint` on your system, make sure you have `npm` installed on your system. To install `stylelint` run the following command:
54
 
55
  ```shell
56
  $ npm i -g stylelint
57
  ```
58
 
59
  > **Note**
60
+ > In the above command the dollar sign(**$**) refers to running the command in privileged mode by using utilities `sudo`, `doas`, `pkgexec`, or any other privileged access methods.
61
 
62
+ - `Cargo-watch` installed on your system which will allow you to auto-build the project when any checks occur in the source code files in the codebase (`websurfx` directory). Before you install `cargo-watch` on your system, make sure you have `cargo` installed on your system. To install `cargo-watch` run the following command:
63
 
64
  ```shell
65
  cargo install cargo-watch
66
  ```
67
 
68
+ - `Git` installed on your system. The installation instructions for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
69
+ - Finally, The latest version of `Docker` is installed on your system which will be used to avoid introducing unexpected issues when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
70
 
71
  > **Note**
72
+ > For **rolling release Linux distributions (distros)**, the above-mentioned required packages except for `stylelint` and `cargo-watch` can also be installed via the distro-specific package manager.
73
  >
74
  > **For Example:**
75
  >
76
+ > On `arch linux` the following packages can be installed by following the link to the installation instructions provided below:
77
  >
78
  > - `Cargo`: https://wiki.archlinux.org/title/rust
79
  > - `Npm`: https://wiki.archlinux.org/title/Node.js
 
81
  > - `Git`: https://wiki.archlinux.org/title/git
82
  > - `Docker`: https://wiki.archlinux.org/title/docker
83
  >
84
+ > But we do not recommend this method for **stable release Linux distros** as they tend to not provide very up-to-date versions of the required packages.
85
 
86
  #### Setting up Pre-commit Checks
87
 
88
+ Before you set `pre-commit` checks, you will first need to clone **your fork of the project** and navigate into the cloned repository by running the following command:
89
 
90
  ```shell
91
  git clone https://github.com/<your_github_username>/websurfx.git
92
  cd websurfx
93
  ```
94
 
95
+ Once you have finished running the above commands then run the following command to set the `pre-commit` checks:
96
 
97
  ```shell
98
  cargo test
99
  ```
100
 
101
+ By running the above-mentioned command, it will automatically set up all the pre-commit checks in the project.
102
 
103
  #### Running the Project
104
 
 
108
  cargo watch -q -x "run" -w "."
109
  ```
110
 
111
+ This will compile the app by default with the **In-Memory caching** feature. To compile, run, and test the app with other features follow the build options listed below:
112
 
113
  ##### Hybrid Cache
114
 
 
140
  > redis-server --port 8082 &
141
  > ```
142
 
143
+ Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to be rebuilt.
144
 
145
  > For more info on all the project commands. See: [**Project Commands**](#project-commands-)
146
 
147
  ### Gitpod
148
 
149
+ This section covers how to use and set up the Gitpod development environment for working on the project.
150
 
151
  > **Note**
152
  > By default the project only supports the Vscode **IDE/Editor** for Gitpod.
 
167
 
168
  The project by default provides a set of pre-installed plugins for gitpod which is done to improve productivity and efficiency while working on the project. Also to make working on the project more fun and engaging which can be customized from within the `Gitpod` instance.
169
 
170
+ The list of all the pre-installed plugins are listed below:
171
 
172
  **Productivity**
173
 
 
191
  - [Test Explorer UI](https://open-vsx.org/extension/hbenl/vscode-test-explorer): Provides a way to run any test from a convenient sidebar.
192
  - [GitLens](https://open-vsx.org/extension/eamodio/gitlens): Provides a better and more efficient way to manage common git workflows.
193
 
194
+ > Optionally, if you prefer a more keyboard-centric workflow then we would recommend using the following extension:
195
  >
196
  > - [VSCode Neovim](https://open-vsx.org/extension/asvetliakov/vscode-neovim): Provides complete vim emulation for vscode.
197
 
 
204
 
205
  ### NixOS Dev Shell using Nix Flake
206
 
207
+ This section covers how to setup the project for development using the `NixOS dev-shell`.
208
 
209
  #### Pre Setup Requirements
210
 
211
  Before you start working on the project. You will need the following packages installed on your system:
212
 
213
+ - `Git` installed on your system. The installation instructions for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
214
+ - Finally, The latest version of `Docker` is installed on your system which will be used to avoid introducing unexpected issues when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
215
 
216
+ > Optionally, On `NixOS` the above-mentioned required packages except for `stylelint` and `cargo-watch` could also be installed by following the link to the installation instructions provided below:
217
  >
218
  > - `Git`: https://search.nixos.org/packages?channel=23.05&show=git&from=0&size=50&sort=relevance&type=packages&query=git
219
  > - `Docker`: https://search.nixos.org/packages?channel=23.05&show=docker&from=0&size=50&sort=relevance&type=packages&query=docker
 
233
  nix develop
234
  ```
235
 
236
+ Once you have finished running the above commands then run the following command to setup the `pre-commit` checks:
237
 
238
  ```shell
239
  cargo test
240
  ```
241
 
242
+ By running the above-mentioned command, it will automatically set up all the pre-commit checks in the project.
243
 
244
  #### Post Setup Requirements
245
 
246
+ After you have done setting up pre-commit checks, then you may need to fulfill a few more requirements to finish setting up the development environment with `NixOS dev-shell`. These include:
247
 
248
+ - `Cargo-watch` installed on your system which will allow you to auto-build the project when any checks occur in the source code files in the codebase (`websurfx` directory). Before you install `cargo-watch` on your system, make sure you have `cargo` installed on your system. To install `cargo-watch` run the following command:
249
 
250
  ```shell
251
  cargo install cargo-watch
 
259
  cargo watch -q -x "run" -w "."
260
  ```
261
 
262
+ This will compile the app by default with the **In-Memory caching** feature. To compile, run, and test the app with other features follow the build options listed below:
263
 
264
  ##### Hybrid Cache
265
 
 
285
  cargo watch -q -x "run --no-default-features --features redis-cache" -w .
286
  ```
287
 
288
+ > Optionally, If you have build and run the app with the `Redis cache`or `Hybrid cache` feature (as mentioned above) then you will need to start the redis server alongside the app which can be done by running the following command:
289
  >
290
  > ```shell
291
  > redis-server --port 8082 &
292
  > ```
293
 
294
+ Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to be rebuilt.
295
 
296
  ### Local Development with Docker Compose
297
 
298
+ This section covers how to set up the project for development on your local machine (bare metal) using `docker compose`.
299
 
300
  #### Prerequisites
301
 
302
  Before you start working on the project. You will need the following packages installed on your system:
303
 
304
+ - The latest version of `cargo` installed on your system which is required to manage the building and running the project. The installation instructions for this can be found [here](https://doc.rust-lang.org/cargo/getting-started/installation.html).
305
+ - The latest version of `npm` installed on your system which is required to allow the installation of other tools necessary for the project. The installation for this can be found [here](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
306
+ - The latest version of `stylelint` should be installed on your system which will be used by the pre-commit checks to lint the code before a commit can be made to ensure better code quality. Before you install `stylelint` on your system, make sure you have `npm` installed on your system. To install `stylelint` run the following command:
307
 
308
  ```shell
309
  $ npm i -g stylelint
310
  ```
311
 
312
  > **Note**
313
+ > In the above command the dollar sign(**$**) refers to running the command in privileged mode by using utilities `sudo`, `doas`, `pkgexec`, or any other privileged access methods.
314
 
315
+ - `Git` installed on your system. The installation instructions for this can be found [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
316
+ - Finally, The latest version of `Docker` is installed on your system which will be used to avoid introducing unexpected issues when working on the project. The installation instructions for this can be found [here](https://docs.docker.com/engine/install/).
317
 
318
  > **Note**
319
+ > For **rolling release Linux distributions (distros)**, the above-mentioned all required packages can also be installed via the distro-specific package manager.
320
  >
321
  > **For Example:**
322
  >
323
+ > On `arch linux` the following packages can be installed by following the link to the installation instructions provided below:
324
  >
325
  > - `Cargo`: https://wiki.archlinux.org/title/rust
326
  > - `Npm`: https://wiki.archlinux.org/title/Node.js
327
  > - `Git`: https://wiki.archlinux.org/title/git
328
  > - `Docker`: https://wiki.archlinux.org/title/docker
329
  >
330
+ > But we do not recommend this method for **stable release Linux distros** as they tend to not provide very up-to-date versions of the required packages.
331
 
332
  #### Setting up Pre-commit Checks
333
 
 
338
  cd websurfx
339
  ```
340
 
341
+ Once you have finished running the above commands then run the following command to setup the `pre-commit` checks:
342
 
343
  ```shell
344
  cargo test
345
  ```
346
 
347
+ By running the above-mentioned command, it will automatically set up all the pre-commit checks in the project.
348
 
349
  #### Running the Project
350
 
 
355
  ```
356
 
357
  > **Note**
358
+ > In the above command the dollar sign(**$**) refers to running the command in privileged mode by using utilities `sudo`, `doas`, `pkgexec`, or any other privileged access methods.
359
 
360
+ Once you have finished running the above command, Websurfx should now be served on the address http://127.0.0.1:8080. Hot reload is enabled, so making changes to any of the files will trigger the project to be rebuilt.
361
 
362
  ### Project Commands
363
 
 
383
 
384
  ### Environment Variables
385
 
386
+ All environment variables are optional. Currently, there are not many environment variables used, as most of the user preferences are stored under the `websurfx` folder (located under the codebase (`websurfx` directory)) in the `config.lua` file.
387
 
388
+ The list of all the available environment variables are listed below:
389
 
390
+ - `PKG_ENV`: Sets the logging level for the app to **Trace** which can be useful for better debugging of the app. These environment variables accept two values `dev` or `prod` as strings.
391
+ - `RUST_BACKTRACE`: Rust-specific environment variable useful for getting more elaborate error messages with an error stack to better diagnose the issue. This environment variable accepts three values `0` (off), `1` (on), and `full` (for long error stack to being printed out).
392
 
393
  ## Git Strategy
394
 
 
396
 
397
  Like most Git repos, we are following the [Github Flow](https://guides.github.com/introduction/flow) standard.
398
 
399
+ 1. Create a branch (or fork if you don't have write access)
400
  2. Code some awesome stuff πŸ§‘β€πŸ’»
401
+ 3. Add, commit, and push your changes to your branch/ fork
402
  4. Head over to GitHub and create a Pull Request
403
  5. Fill in the required sections in the template, and hit submit
404
  6. Follow up with any reviews on your code
 
411
 
412
  ### Commit Emojis
413
 
414
+ Using a single emoji at the start of each commit message, issue title, and pull request title, to indicate the type of task, makes the commit ledger, issue, and pull request easier to understand, it looks cool.
415
 
416
+ - 🎨 `:art:` - Improve the structure/format of the code.
417
  - ⚑️ `:zap:` - Improve performance.
418
  - πŸ”₯ `:fire:` - Remove code or files.
419
  - πŸ› `:bug:` - Fix a bug.
 
426
  - βœ… `:white_check_mark:` - Add, update, or pass tests.
427
  - πŸ”’οΈ `:lock:` - Fix security issues.
428
  - πŸ”– `:bookmark:` - Make a Release or Version tag.
429
+ - 🚨 `:rotating_light:` - Fix compiler/linter warnings.
430
  - 🚧 `:construction:` - Work in progress.
431
  - ⬆️ `:arrow_up:` - Upgrade dependencies.
432
+ - πŸ‘· `:construction_worker:` - Add or update the CI build system.
433
  - ♻️ `:recycle:` - Refactor code.
434
  - 🩹 `:adhesive_bandage:` - Simple fix for a non-critical issue.
435
  - πŸ”§ `:wrench:` - Add or update configuration files.
436
  - 🍱 `:bento:` - Add or update assets.
437
+ - πŸ—ƒοΈ `:card_file_box:` - Perform database schema-related changes.
438
  - ✏️ `:pencil2:` - Fix typos.
439
  - 🌐 `:globe_with_meridians:` - Internationalization and translations.
440
 
 
446
 
447
  For a pull request to be merged, it must:
448
 
449
+ - The build, lint, and tests (run by GH actions) must pass
450
  - There must not be any merge conflicts
451
 
452
  When you submit your pull request, include the required info, by filling out the pull request template. Including:
453
 
454
  - A brief description of your changes.
455
  - The issue or ticket number (if applicable).
456
+ - For UI-related updates include a screenshot.
457
+ - If any dependencies were added, explain why it was needed, and state the cost. associated, and confirm it does not introduce any security, privacy, or speed issues
458
+ - Optionally, provide a checklist of all the changes that were included in the pull request.
459
 
460
  > **Important**
461
+ > Make sure to fill all the required/mandatory sections of the pull request as filling them helps us distinguish between spam pull requests and legitimate pull requests.
462
 
463
  > **Note**
464
  > The pull request template contains comments in the following form `<!-- -->` which are used to provide a guide on what should be provided under each heading of the template. These comments are never rendered when the pull request is either created or updated and hence anything provided in such comments is never displayed.
465
 
466
  ## Resources for Beginners
467
 
468
+ New to Web Development? Or New to GitHub? Glad to see you're here!! :slightly_smiling_face: Websurfx is a pretty simple app, so it should make a good candidate for your first PR. The following articles (which have been divided into parts for convenience) should point you in the right direction for getting up to speed with the technologies used in this project:
469
 
470
  **Development**
471
 
 
489
  - [A Tutorial on Git](https://piped.kavin.rocks/playlist?list=PL4lTrYcDuAfxAgSefXftJXbhw0qvjfOFo)
490
  - [Git cheat sheet](http://git-cheatsheet.com/)
491
 
492
+ For Rust, CSS, JS, HTML, Git, and Docker- you'll need an IDE (e.g. [VSCode](https://code.visualstudio.com/) or [Neovim](https://neovim.io/) and a terminal (Windows users may find [WSL](https://docs.microsoft.com/en-us/windows/wsl/) more convenient).
493
 
494
  ## App Info
495
 
 
509
  ## Application Structure
510
 
511
  > **Important**
512
+ > We follow the Unix style naming conventions for all the files and folders in the project (except for all files under the `themes` and `colorschemes` folder in the frontend's source code which requires that the names of the files and folders should be in lowercase and the words be separated with a hyphen.) which includes the name of the files and folders should be in lowercase and every word should be separated with an underscore.
513
 
514
  **Files in the root of the codebase:** `./`
515
 
 
543
  β”œβ”€β”€ static # The directory containing all the UI handlers.
544
  β”‚ β”œβ”€β”€ cookies.js # Handles the loading of saved cookies.
545
  β”‚ β”œβ”€β”€ error_box.js # Handles the toggling functionality of the error box on the search page.
546
+ β”‚ β”œβ”€β”€ index.js # Functions to handle the search functionality of the search bar.
547
+ β”‚ β”œβ”€β”€ pagination.js # Functions to handle the navigation between the previous and next page in the search page.
548
+ β”‚ β”œβ”€β”€ search_area_options.js # Changes the search options under the search bar in the search page according to the safe search level set using the URL safesearch parameter.
549
+ β”‚ β”œβ”€β”€ settings.js # Handles the settings and saving of all the settings page options as a cookie.
550
  β”‚ β”œβ”€β”€ colorschemes # A folder containing all the popular colorscheme files as CSS files.
551
  β”‚ └── themes # A folder containing all the popular theme files as CSS files.
552
  └── templates # Folder containing all the template files for the different pages on the website.
553
+ β”œβ”€β”€ 404.html # A 404-page template.
554
  β”œβ”€β”€ about.html # An about page template.
555
  β”œβ”€β”€ bar.html # A template for the search bar.
556
  β”œβ”€β”€ cookies_tab.html # A template for the cookies tab for the settings page.
557
  β”œβ”€β”€ engines_tab.html # A template for the engines tab for the settings page.
558
+ β”œβ”€β”€ footer.html # A footer template for all pages.
559
  β”œβ”€β”€ general_tab.html # A template for the general tab for the settings page.
560
+ β”œβ”€β”€ header.html # A header template for all pages.
561
  β”œβ”€β”€ index.html # A home page template.
562
  β”œβ”€β”€ navbar.html # A navbar template for the header template.
563
  β”œβ”€β”€ search.html # A search page template.
 
572
  ./src/
573
  β”œβ”€β”€ lib.rs # A library file for the rust project.
574
  β”œβ”€β”€ bin # A folder containing the source code that would produce the binary file when compiled.
575
+ β”‚ └── websurfx.rs # A file that would be compiled into a binary file.
576
  β”œβ”€β”€ cache # A folder that contains code to handle the caching functionality of the search engine.
577
  β”‚ β”œβ”€β”€ cacher.rs # Handles the different caching features.
578
  β”‚ β”œβ”€β”€ error.rs # Provides custom error messages for different types of caches and their related errors.
 
587
  β”‚ β”œβ”€β”€ mod.rs # A module file for the rust project.
588
  β”‚ β”œβ”€β”€ search_result_parser.rs # Provides helper function to help ease the process of defining different result selection selectors.
589
  β”‚ └── searx.rs # Provides code to fetch and remove unnecessary or waste results from the fetched results from the searx engine.
590
+ β”œβ”€β”€ handler # A folder that provides helper code to provide a proper path to the public (theme) folder, config file, blocklist file, and allowlist file based on where they are located.
591
  β”‚ β”œβ”€β”€ mod.rs # A module file for the rust project.
592
  β”‚ └── paths.rs # Provides helper code to handle different paths.
593
  β”œβ”€β”€ models # A folder that provides different models for the different modules in the backend code.
 
596
  β”‚ β”œβ”€β”€ mod.rs # A module file for the rust project.
597
  β”‚ β”œβ”€β”€ parser_models.rs # Provides different models (enums, structs) for handling and standardizing different parts in the "config" module code.
598
  β”‚ └── server_models.rs # Provides different models (enums, structs) for handling and standardizing different parts in the "server" module code.
599
+ β”œβ”€β”€ results # A folder that provides code to handle the fetching and aggregating of results from the upstream search engines.
600
+ β”‚ β”œβ”€β”€ aggregator.rs # Provides code aggregate and fetches results from the upstream engines.
601
  β”‚ β”œβ”€β”€ mod.rs # A module file for the rust project.
602
+ β”‚ └── user_agent.rs # Provides a helper function to allow random user agents to pass in the server request code to improve user privacy and avoiding detected as a bot.
603
  └── server # A folder that holds code to handle the routes for the search engine website.
604
  β”œβ”€β”€ mod.rs # A module file for the rust project.
605
  β”œβ”€β”€ router.rs # Provides functions to handle the different routes on the website.
 
612
 
613
  ### Performance - Lighthouse
614
 
615
+ The easiest method of checking performance is to use Chromium's built-in auditing tool, Lighthouse. To run the test, open Developer Tools (usually F12) --> Lighthouse and click on the 'Generate Report' button at the bottom.
616
 
617
  ## Notes
618
 
619
  ### Known warnings
620
 
621
+ When running the build command, a warning appears. This is not an error and does not affect the security or performance of the application. They will be addressed soon in a future update.
622
 
623
  ```shell
624
  warning: the following packages contain code that will be rejected by a future version of Rust: html5ever v0.23.0
625
  note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 2`
626
  ```
627
 
628
+ This warning just means that any dependencies or code using the `html5ever` code would be deprecated and rejected in future versions of the Rust language. So right now these dependencies can be used as these have not happened yet.
629
 
630
  [⬅️ Go back to Home](./README.md)