NodeBB

NodeBB

Member Since 8 years ago

Toronto, ON

Experience Points
0
follower
Lessons Completed
0
follow
Best Reply Awards
76
repos
Activity
Oct
17
1 day ago
Activity icon
created branch

renovate[bot] in NodeBB/NodeBB create branch renovate/nodebb-theme-persona-11.x

createdAt 21 hours ago
Activity icon
created tag
createdAt 21 hours ago
Activity icon
issue

barisusakli issue NodeBB/NodeBB

barisusakli
barisusakli

search and chat dropdowns are unreadable on darkly skin

Activity icon
issue

barisusakli issue NodeBB/NodeBB

barisusakli
barisusakli

notifications and chat dropdowns are unreadable on darkly skin

started
started time in 22 hours ago
Activity icon
issue

sadaszewski issue comment NodeBB/NodeBB

sadaszewski
sadaszewski

Transactionality / consistency

Hi, This is more of an architecture / understanding question than a bug report. Is it correct to say that NodeBB can find itself in an inconsistent state if it crashes? For example, it is possible that only a part of the housekeeping logic related to creating a post is executed. It can happen each time the application is killed in the middle of performing an action. This is due to the fact that there is no attempt to formulate the logic in a transactional/crash-proof manner. If so, are there any plans to change that? One good solution would be to run a type of periodic housekeeping activity that would run on a global mutex, locking out any other processing and making an atomic snapshot of the database. The database would have to support doing that efficiently. Like this, we could have the confidence that we are in a consistent state (by restoring a snapshot rather than using the "live" version after a crash) and the overhead would not necessarily be prohibitive, depending on the method used to snapshot the DB. What do you think?

sadaszewski
sadaszewski

@pitaj this is where it gets quite interesting. In SQL-based drivers this could be achieved (albeit with some overhead) with simple [valid_from, valid_to] timestamping of all rows in all tables. This approach is a well-known practice among the SQL old-timers and it allows to access the precise state of a database at any point in time by simply including a timestamp in all queries. Restoring the state from any timepoint is also possible by simply deleting all rows with valid_from timestamps older than the desired one.

In the above case "making a snapshot" would mean only remembering a certain timestamp as "reliable" (i.e. from a moment when NodeBB was not doing anything to the DB). This approach should be fast and robust with appropriate indexing.

Alternatively, for all DBs there is the option of filesystem-level snapshots. I am a FreeBSD user and thus am very spoiled since many many years with how easy it is to take snapshots on ZFS. Having a hook in NodeBB which would be called when everything is guaranteed to be flushed to the DB and idle would allow to do the job by calling an external application (a bash script) and waiting for its termination. The external process would take the snapshot. One could take care of restoring it in a custom startup script. In this scenario the work on NodeBB should be pretty minimal.

With regards to stopping the database, it would not be an issue with the above 2 approaches (i.e. not necessary in the first case and relatively easy to do in the second). In the second case though it would be necessary for NodeBB to reconnect. This could be part of the hook function.

pull request

oplik0 pull request NodeBB/docs

oplik0
oplik0

Some version updates and deprecated service removal

A general package of version bumps and updates to documentation:

  • bumps NodeBB version across installation and upgrade instructions
  • bumps Debian version to Buster (10) and Ubuntu to Focal Fossa (20.04)
  • bumps recommended MongoDB version to 5.0 and minimum to 3.2 (older aren't supported by connect-mongo)
  • bumps minimum Node version to 12
  • bumps minimum redis version to 2.6.12 (older aren't supported by ioredis)
  • updates mongodb installation and configuration instructions
    • including removal of pre-2.6 instructions, since 3.2 is the minimum anyway
  • updates ./nodebb command output to current one
  • removes koding.com instructions, as the hosted solution shut down in 2018 (and the self-hosted one isn't maintained since then too with the only update since being a documentation fix)

These updates were largely motivated by seeing someone having problems with a "new" NodeBB v1.10 install due to old docs on the community forum - just got around to it now because I was working on NodeBB anyway :)

Activity icon
fork

oplik0 forked NodeBB/docs

⚡ NodeBB Documentation via MkDocs
oplik0 MIT License Updated
fork time in 1 day ago
Activity icon
issue

oplik0 issue comment NodeBB/NodeBB

oplik0
oplik0

Transactionality / consistency

Hi, This is more of an architecture / understanding question than a bug report. Is it correct to say that NodeBB can find itself in an inconsistent state if it crashes? For example, it is possible that only a part of the housekeeping logic related to creating a post is executed. It can happen each time the application is killed in the middle of performing an action. This is due to the fact that there is no attempt to formulate the logic in a transactional/crash-proof manner. If so, are there any plans to change that? One good solution would be to run a type of periodic housekeeping activity that would run on a global mutex, locking out any other processing and making an atomic snapshot of the database. The database would have to support doing that efficiently. Like this, we could have the confidence that we are in a consistent state (by restoring a snapshot rather than using the "live" version after a crash) and the overhead would not necessarily be prohibitive, depending on the method used to snapshot the DB. What do you think?

oplik0
oplik0

I think only redis has built-in snapshot support, and that's because it's an in-memory database with snapshots as one of two persistence method (with the more basic being AOF that saves every operation instead of doing so only on specified intervals or manually).

However even there restoring snapshots requires stopping the database.

Activity icon
issue

pitaj issue comment NodeBB/NodeBB

pitaj
pitaj

Transactionality / consistency

Hi, This is more of an architecture / understanding question than a bug report. Is it correct to say that NodeBB can find itself in an inconsistent state if it crashes? For example, it is possible that only a part of the housekeeping logic related to creating a post is executed. It can happen each time the application is killed in the middle of performing an action. This is due to the fact that there is no attempt to formulate the logic in a transactional/crash-proof manner. If so, are there any plans to change that? One good solution would be to run a type of periodic housekeeping activity that would run on a global mutex, locking out any other processing and making an atomic snapshot of the database. The database would have to support doing that efficiently. Like this, we could have the confidence that we are in a consistent state (by restoring a snapshot rather than using the "live" version after a crash) and the overhead would not necessarily be prohibitive, depending on the method used to snapshot the DB. What do you think?

pitaj
pitaj

Do all of the databases support snapshots? How long do they take? Do they require pausing everything for the duration? It would be bad if the forum totally paused for seconds at a time.

Activity icon
issue

sadaszewski issue comment NodeBB/NodeBB

sadaszewski
sadaszewski

Transactionality / consistency

Hi, This is more of an architecture / understanding question than a bug report. Is it correct to say that NodeBB can find itself in an inconsistent state if it crashes? For example, it is possible that only a part of the housekeeping logic related to creating a post is executed. It can happen each time the application is killed in the middle of performing an action. This is due to the fact that there is no attempt to formulate the logic in a transactional/crash-proof manner. If so, are there any plans to change that? One good solution would be to run a type of periodic housekeeping activity that would run on a global mutex, locking out any other processing and making an atomic snapshot of the database. The database would have to support doing that efficiently. Like this, we could have the confidence that we are in a consistent state (by restoring a snapshot rather than using the "live" version after a crash) and the overhead would not necessarily be prohibitive, depending on the method used to snapshot the DB. What do you think?

sadaszewski
sadaszewski

@oplik0 nice, thanks for that insight! so do you think an approach like this could work:

  1. introduce dbCounter - a global counter of things in progress that use the DB i. these are for sure the requests to the Express app, what else?
  2. introduce dbLock - a variable saying that the DB is locked (for the duration of a snapshot)
  3. add new middleware to the Express app which would: i. if dbLock is set, push the next middleware to a waiting list and exit ii. if dbLock is not set:
  • increment the global counter
  • await the next middleware
  • decrement the global counter
  1. run a timer (e.g. every minute) which would: i. check if dbCounter is zero - if not, do nothing and exit ii. if dbCounter is zero, set dbLock and: iii. make a snapshot of the database (implemented by the DB driver) iv. unset dbLock v. for each element in the waiting list, start a Promise, which:
  • increments the dbCounter
  • awaits the stored next middleware handler
  • decrements the dbCounter

Add one more step upon shutdown:

  • after all is idle, make a snapshot of the database

Them upon startup add one more step:

  • restore the database from the last snapshot

Like this, if a crash occurs, one would always be able to restore the last fully written snapshot. What do you say?

Activity icon
issue

joolswills issue NodeBB/NodeBB

joolswills
joolswills

Topics show as "read" for Logged out / Unregistered

  • NodeBB version: v1.18.2
  • NodeBB git hash: 8593ea87e9b 5d60eccd6e101d86b4f4e3a5ef4b3
  • **NodeJS version:**14.18.1-
  • Exact steps to cause this issue: View topics in recent / or via categories when not logged in or unregistered.
  • What you expected: Topics should look like "unread" topics rather than grey
  • What happened instead: Topics show as grey / read rather than the unread colour for logged in users.

I'm sure this never used to be the case, but I would have thought it would be more logical for topics to show as the unread colour when not logged in or a guest comes to the forum.

Activity icon
issue

oplik0 issue comment NodeBB/NodeBB

oplik0
oplik0

Transactionality / consistency

Hi, This is more of an architecture / understanding question than a bug report. Is it correct to say that NodeBB can find itself in an inconsistent state if it crashes? For example, it is possible that only a part of the housekeeping logic related to creating a post is executed. It can happen each time the application is killed in the middle of performing an action. This is due to the fact that there is no attempt to formulate the logic in a transactional/crash-proof manner. If so, are there any plans to change that? One good solution would be to run a type of periodic housekeeping activity that would run on a global mutex, locking out any other processing and making an atomic snapshot of the database. The database would have to support doing that efficiently. Like this, we could have the confidence that we are in a consistent state (by restoring a snapshot rather than using the "live" version after a crash) and the overhead would not necessarily be prohibitive, depending on the method used to snapshot the DB. What do you think?

oplik0
oplik0

While some exceptions may be caught, uncaught errrors do crash NodeBB. However, IIRC the crash should result in a restart normally and only issues during startup will actually shutdown NodeBB.

pull request

oplik0 pull request NodeBB/NodeBB

oplik0
oplik0

Use MongoDB transactions

I mentioned that it'd be quite simple to add MongoDB transactions in #9898 - so I decided to try it, even if it's not ideal when it comes to consistency.

I think I handled all parts of MongoDB driver where multiple writes occur (since single operations are atomic anyway).

Transactions only work on replica sets with a version newer than MongoDB 4.0, unfortunately. On singular instances nothing improves in terms of consistency. As such the current tests never take the transaction path - and to change that would require adding another entry to test matrix for newer replica set MongoDB. If it's fine to do so, I'll be glad to update this PR to have testing done properly for replica sets, but currently I just made a separate branch with all mongo tests modified to use replica sets: https://github.com/oplik0/NodeBB/actions/runs/1351224224

push

nodebb-misty push NodeBB/NodeBB

nodebb-misty
nodebb-misty

Latest translations and fallbacks

commit sha: 1339c75781f7679f57645499a44003915719b901

push time in 1 day ago
started
started time in 1 day ago
Activity icon
issue

sadaszewski issue comment NodeBB/NodeBB

sadaszewski
sadaszewski

Transactionality / consistency

Hi, This is more of an architecture / understanding question than a bug report. Is it correct to say that NodeBB can find itself in an inconsistent state if it crashes? For example, it is possible that only a part of the housekeeping logic related to creating a post is executed. It can happen each time the application is killed in the middle of performing an action. This is due to the fact that there is no attempt to formulate the logic in a transactional/crash-proof manner. If so, are there any plans to change that? One good solution would be to run a type of periodic housekeeping activity that would run on a global mutex, locking out any other processing and making an atomic snapshot of the database. The database would have to support doing that efficiently. Like this, we could have the confidence that we are in a consistent state (by restoring a snapshot rather than using the "live" version after a crash) and the overhead would not necessarily be prohibitive, depending on the method used to snapshot the DB. What do you think?

sadaszewski
sadaszewski

@oplik0 thanks for the summary. Sounds like all the more reason to seek a workaround in the meantime. Do you know if the application crashes when there are exceptions in the request handlers? Or are they silently ignored?

started
started time in 1 day ago
push

renovate[bot] push NodeBB/NodeBB

renovate[bot]
renovate[bot]

fix(deps): update dependency socket.io-client to v4.3.2

commit sha: deba3e27521ccbbae5ed9d5e1568d1d822a8714a

push time in 1 day ago
Activity icon
delete

renovate[bot] in NodeBB/NodeBB delete branch renovate/socket.io-packages

deleted time in 1 day ago
pull request

renovate[bot] pull request NodeBB/NodeBB

renovate[bot]
renovate[bot]

fix(deps): update dependency socket.io-client to v4.3.2

WhiteSource Renovate

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
socket.io-client 4.3.1 -> 4.3.2 age adoption passing confidence

Release Notes

socketio/socket.io-client

v4.3.2

Compare Source

Bug Fixes
  • restore the default export (bis) (6780f29)

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Enabled.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box.

This PR has been generated by WhiteSource Renovate. View repository job log here.

pull request

renovate[bot] pull request NodeBB/NodeBB

renovate[bot]
renovate[bot]

fix(deps): update dependency socket.io-client to v4.3.2

WhiteSource Renovate

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
socket.io-client 4.3.1 -> 4.3.2 age adoption passing confidence

Release Notes

socketio/socket.io-client

v4.3.2

Compare Source

Bug Fixes
  • restore the default export (bis) (6780f29)

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Enabled.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box.

This PR has been generated by WhiteSource Renovate. View repository job log here.

Activity icon
created branch

renovate[bot] in NodeBB/NodeBB create branch renovate/socket.io-packages

createdAt 1 day ago
Oct
16
2 days ago
push

renovate[bot] push NodeBB/NodeBB

renovate[bot]
renovate[bot]

fix(deps): update dependency socket.io to v4.3.1

commit sha: e1554f619abb8b09afa63a17ddedbf563db28aef

push time in 1 day ago
Activity icon
delete

renovate[bot] in NodeBB/NodeBB delete branch renovate/socket.io-packages

deleted time in 1 day ago
pull request

renovate[bot] pull request NodeBB/NodeBB

renovate[bot]
renovate[bot]

fix(deps): update dependency socket.io to v4.3.1

WhiteSource Renovate

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
socket.io 4.3.0 -> 4.3.1 age adoption passing confidence

Release Notes

socketio/socket.io

v4.3.1

Compare Source

Bug Fixes

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Enabled.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box.

This PR has been generated by WhiteSource Renovate. View repository job log here.

pull request

renovate[bot] pull request NodeBB/NodeBB

renovate[bot]
renovate[bot]

fix(deps): update dependency socket.io to v4.3.1

WhiteSource Renovate

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
socket.io 4.3.0 -> 4.3.1 age adoption passing confidence

Release Notes

socketio/socket.io

v4.3.1

Compare Source

Bug Fixes

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Enabled.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box.

This PR has been generated by WhiteSource Renovate. View repository job log here.

Activity icon
created branch

renovate[bot] in NodeBB/NodeBB create branch renovate/socket.io-packages

createdAt 1 day ago