KN-1011 - 4.0.A.13 - Booting @Fixed Time

[KYTECHNGAMING]

Hello,

I am using KN-1011-EU.

The other day, I noticed that the system uptime and WAN IP are different when WG tunnel is not working. I did a check last night.
One day after installing 4.0.A.13, the main router would reboot at the same time every night.

03:21 AM There is a reboot action for 3 nights.
It was 5:30 AM when I took the self-test last night.

Shared privately.

Mobile App Events:
https://prnt.sc/UdtNjez27gl_ https://prnt.sc/9h6omMLjKT4K https://prnt.sc/1C1yJbbdY9F1

Logs:
https://prnt.sc/fDi3BAsl_Hn5 https://prnt.sc/0kVjoZbKXors (SOC power status - not sure if it shows physical or software.)

 

 

Изменено 14 марта, 2023 пользователем KYTECHNGAMING
Change SS

[KYTECHNGAMING]

Could it be triggered by the server for the EU variant?
In one of the last updates, I was asked to choose a region and I did it.
Just a thought.

[KYTECHNGAMING]

The reboot situation also occurred tonight.

https://prnt.sc/aLmCxfMNCS6k

 

[sergeyk]

@KYTECHNGAMING looks like the root cause of reboots is a tsmb module crash (see proc:mtdoops/ndm section in your self-tests).

"oops": { 
"origin": "kernel",
"version":"4.00.A.13.0-0",
"board": "KN-1011",
"timestamp": "2023-03-14 00:19:58",
"hash": "b6d5955",

and

"oops": { 
"origin": "kernel",
"version":"4.00.A.13.0-0",
"board": "KN-1011",
"timestamp": "2023-03-15 00:19:59",
"hash": "9e603c3a",

Could you investigate what happens at this time in your network? I do not see any configured schedules in your startup configuration.

Probably a packet capture for a SMB protocol close to this time will help (a packet capture should be stored on a USB storage to keep it in a case of a kernel crash).

[KYTECHNGAMING]

23 minutes ago, sergeyk said:

@KYTECHNGAMING looks like the root cause of reboots is a tsmb module crash (see proc:mtdoops/ndm section in your self-tests).

"oops": { 
"origin": "kernel",
"version":"4.00.A.13.0-0",
"board": "KN-1011",
"timestamp": "2023-03-14 00:19:58",
"hash": "b6d5955",

and

"oops": { 
"origin": "kernel",
"version":"4.00.A.13.0-0",
"board": "KN-1011",
"timestamp": "2023-03-15 00:19:59",
"hash": "9e603c3a",

Could you investigate what happens at this time in your network? I do not see any configured schedules in your startup configuration.

Probably a packet capture for a TSMB protocol close to this time will help (a packet capture should be stored on a USB storage to keep it in a case of a kernel crash).

Hello sergeyk,

Yes, I don't have any scheduler.

It has been going on for 4 nights.
I'm going to open syslog today.

According to your findings; At that hour, I think it might just be my Mi camera transferring its backup to disk.

[sergeyk]

4 минуты назад, KYTECHNGAMING сказал:

I think it might just be my Mi camera transferring its backup to disk.

I suppose you may reconfigure the camera to make backups at any time to speedup the investigation.

[KYTECHNGAMING]

1 minute ago, sergeyk said:

I suppose You may reconfigure the camera to make backups at any time to speedup the process.

Thanks to you, we got closer to the source of the problem. I didn't set it to sync to this time.But his latest backup seems to be stuck here too.
 

https://prnt.sc/OWUfADL6u5Ba

 

 

 

[KYTECHNGAMING]

It's a really interesting situation.
2 different Mi cameras. They have different synchronous scheduling and write data to the same disk.
One of them stopped writing. Could the problem be with the disk? Did the bad sector coincide with the area? It also keeps recording on internal sd-card.

1. camera https://prnt.sc/EyiXkLgQtZ2D

2. camera https://prnt.sc/07BZ-FJVM6MZ

[KYTECHNGAMING]

In the previous message, I showed that the 1st camera was stuck in the past date and time.. @sergeyk
I set the sync to "Live" to see if we can trigger the issue.

At 13:20 the syslog record was captured. Router restarted. (Also, no recording has been transferred to my disk.)

https://prnt.sc/Mng0ahPTdf-a

Time: Mar 15 13:20:00
IP: 10.101.0.1
Host: Keenetic-8178
Facility: kern
Priority: crit
Tag: kernel
Message: CPU 0 Unable to handle kernel paging request at virtual address 00000000, epc == 9ac62b40, ra == 9ac523c0

Time: Mar 15 13:20:00
IP: 10.101.0.1
Host: Keenetic-8178
Facility: kern
Priority: warning
Tag: kernel
Message: Oops[#1]:

Time: Mar 15 13:20:00
IP: 10.101.0.1
Host: Keenetic-8178
Facility: kern
Priority: warning
Tag: kernel
Message: CPU: 0 PID: 718 Comm: vfs-ctrl-650e00 Tainted: P           O    4.9-ndm-5 #0

Why is the date and time sorting messed up in syslog?
log time: https://prnt.sc/h9StW35odgvG https://prnt.sc/le1G9a2n4VW5

[sergeyk]

1 час назад, KYTECHNGAMING сказал:

At 13:20 the syslog record was captured. Router restarted. (Also, no recording has been transferred to my disk.

Unfortunately syslog messages are not enough. It is need the packet dump saved at the crash moment.

Could you use the tcpdump utility in an unbuffered mode from opkg?

tcpdump -U -w /<path-to-usb-storage/tsmb_crash.pcap port 445 >/dev/null 2>&1

 

[KYTECHNGAMING]

40 minutes ago, sergeyk said:

Unfortunately syslog messages are not enough. It is need the packet dump saved at the crash moment.

Could you use the tcpdump utility in an unbuffered mode from opkg?

tcpdump -U -w /<path-to-usb-storage/tsmb_crash.pcap port 445 >/dev/null 2>&1

 

tcpdump installed.
I started the command as follows. https://prnt.sc/XmgjzXZOa7nk

The .pcap file was created on disk, but its content is empty. Is he listening now? Isn't it this way?

[KYTECHNGAMING]

20:20 reboot again. But I guess tcpdump did not work. I could not reach any output. 

 

Just log. https://prnt.sc/9GwzfYBkZW_F

 

 

[sergeyk]

@KYTECHNGAMING Sorry for inconvenience, the right command is

tcpdump -i br0 -s 0 -U -w /<path-to-usb-storage/tsmb_crash.pcap port 445 >/dev/null 2>&1

 

[KYTECHNGAMING]

36 minutes ago, sergeyk said:

@KYTECHNGAMING Sorry for inconvenience, the right command is

tcpdump -i br0 -s 0 -U -w /<path-to-usb-storage/tsmb_crash.pcap port 445 >/dev/null 2>&1

 

Thanks, I will do. 

Reboot again 40 mins ago.

Capturing : https://prnt.sc/H0pgrfgwAfMI

Изменено 15 марта, 2023 пользователем KYTECHNGAMING

[KYTECHNGAMING]

@sergeyk  Rebooted again at 22:20

Send pcap file to you? https://prnt.sc/gqB4LwdG8pQu

Upload to yandex?

[sergeyk]

1 минуту назад, KYTECHNGAMING сказал:

Send pcap file to you?

Please attach here in a hidden post.

[KYTECHNGAMING]

1 minute ago, sergeyk said:

Please attach here in a hidden post.

it is 600mb

[sergeyk]

7 минут назад, KYTECHNGAMING сказал:

it is 600mb

Yes, then please upload to Yandex.

[KYTECHNGAMING]

Just now, sergeyk said:

Yes, then please upload to Yandex.

Send link via hidden message in 30 seconds. 

[sergeyk]

@KYTECHNGAMING please also attach a (hidden) self-test from the running device.

[KYTECHNGAMING]

3 minutes ago, sergeyk said:

@KYTECHNGAMING please also attach a (hidden) self-test from the running device.

As soos as possible 🙏

[slomblobov]

@KYTECHNGAMING Could you provide more information about the camera? Model, firmware version...
This may be helpful in reproducing this issue.

[KYTECHNGAMING]

1 hour ago, vst said:

@KYTECHNGAMING Could you provide more information about the camera? Model, firmware version...
This may be helpful in reproducing this issue.

Hello @vst

I use two of this model. 

Xiaomi Mi 360° Home Security Camera 2K (MJSXJ09CM) Link
FW: 4.1.6_0310

Изменено 16 марта, 2023 пользователем KYTECHNGAMING

[slomblobov]

@KYTECHNGAMING Thank you for gathering diagnostic information! Ticket SYS-800 has been created. All necessary information is provided to the TSMB service provider.

We are waiting for answer.

It would be reasonable to check if the issue is reproduced when ACL component is not installed.

 

[KYTECHNGAMING]

On 3/20/2023 at 2:49 PM, vst said:

@KYTECHNGAMING Thank you for gathering diagnostic information! Ticket SYS-800 has been created. All necessary information is provided to the TSMB service provider.

We are waiting for answer.

It would be reasonable to check if the issue is reproduced when ACL component is not installed.

 

Hello,

I've been making observations for the problem for 1-2 days. And I thought I didn't write some things.

I'm accessing the network disk for file transfer to mi cameras with a different user in keenetic. (I think you already wrote ACL because of this.)
This user only has SMB privileges and permissions to that disk/folder. (Are you suggesting that I access it with admin?)

There are two of these cameras. Both create separate folder tree. 
The settings are exactly the same. Only the backup syncs are different. Hourly vs Daily 
Same sdcards mounted. There is no problem recording to SDcard. Although the technical data writes that the support is maximum 32gb sdcard, 128gb sdcard works.
They are connected to the same repeater on my Keenetic Mesh system. https://prnt.sc/Qm9tgBmcwmwh

One of them continues the backup to the same disk properly..(Misak) 
The other one has already stop the backup along with the reboot issue.(Lal) See the dates of the files in the most recent folders in SMB. (2. folder @ x-plore video) https://disk.yandex.com.tr/i/tKNSVv5NRjht3Q

I turned off a feature in the Mi settings about 48 hours ago. He's not making a backup. Anyway, I don't understand exactly what the setting says, either.  https://disk.yandex.com.tr/i/R9gQ8s2JjNiZJA

But the restart problem seems to have disappeared now.

- I'm not sure, but the camera that created this reboot problem also reboots itself. Just like I couldn't access the videos on the sdcard while recording the phone screen, while writing this article, I noticed that Keenetic was offline in the device list and online after a while.

- The screen recordings also recorded music in the background. Don't forget to turn down the video volume.

 

 

 

Изменено 21 марта, 2023 пользователем KYTECHNGAMING

[KYTECHNGAMING]

🧿 https://prnt.sc/WrZydaHx0wKw

Reboot again  

[slomblobov]

@KYTECHNGAMING Does this issue reproduce when only one camera is backing up?

[KYTECHNGAMING]

24 minutes ago, vst said:

@KYTECHNGAMING Does this issue reproduce when only one camera is backing up?

Hi @vst

I didn't make any setting changes. I upgraded to Keenetic 4.0.A.15.
I can say "yes" to the answer to your question. Dailybackup is active. It reboots once a day in xx:20 time zone.

On the other hand, the other camera transfers hourly backups via smb.

- There is a separate user authorization for SMB access. (I think you asked me to try without ACL in your previous message. So should I give folder permission with the admin user? If you get back to me about this, I can also perform the tests you want.)

[slomblobov]

7 minutes ago, KYTECHNGAMING said:

So should I give folder permission with the admin user? If you get back to me about this, I can also perform the tests you want.)

Please uninstall an ACL component. When this component is not installed all ACL settings are ignored, so you don't need to tune them.

After a test you may be able to install ACL again without losing the settings.

[KYTECHNGAMING]

components remove acl 
components commit

ok. test is starting.

[KYTECHNGAMING]

@vst 
After disabling the ACL, no reboot issue was observed. (26th hour)

https://prnt.sc/BxUxoangqiRT https://prnt.sc/n3zoTANRZVlj

 

 

 

Изменено 29 марта, 2023 пользователем KYTECHNGAMING