I don't understand why you are having a problem.
The computer (at home) should be connected by Ethernet to your router, which you would have port forwarded as necessary. If you port forward port 22 (SSH) that sounds as though it should work. You can even disable the wireless network from the router.
For the phone (away from home), use mobile Internet over the cellular network. That way neither device will be using WiFi.
Set the computer on a static address on your internal network, and if your public IP address is dynamic, go with a DDNS service such as noip.com (or one of the other DDNS services). That way you will have a fixed URL to you home.
I hope this helps.
Following ratter_of_the_shire's comment. I had the question confused. The remarks about Port Forwarding and "without WiFi" in the question threw me.
I guess you could enable the phone's WiFi hotspot, without mobile Internet being enabled, then connect the Linux box to the hotspot. If you know the computer's IP address, then you should be able to open a SSH connection to it with a suitable App on the phone. I have done this the other way round, where I have downloaded files from my phone to my laptop using the WiFi File Transfer App on the phone and a browser on my laptop. However, I don't have WiFi on my Linux systems so I have been unable to try connecting in the set-up you describe.
Sorry the original part of the answer was wrong. I'm leaving it in place in case it is useful to others.